nf-core/mag: a best-practice pipeline for metagenome hybrid assembly and binning.
The main results reproduced: recomputed values matched the published ones within tolerance.
- ✓Same input data as the authors
- ✓No relevant deviation in data/preprocessing
- ✓No authors-side cause for any deviation
- ✓Any deviation was negligible
- 🟡Reported values were only indirectly comparable
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
PARTIALLY REPRODUCED. The study's primary computational claims regarding the nf-core/mag pipeline's ability to increase assembly size, N50, and the number of MAGs through hybrid and group-wise co-assembly were not independently verified due to uncheckable results. However, the data deposit was successfully reproduced, matching exactly with the reported structure of 8 samples and 24 FASTQ files available on Zenodo, confirming the presence of a 791-genome community. While the pipeline's end-to-end execution on the published data could not be independently confirmed, the data's accessibility and structure were consistent with the authors' descriptions. Thus, while the core computational claims remain unverified, the data availability and structure were reproduced as reported.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 56assessed: 2026-06-19 ⛓ 6ca1aa64bb99
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-19
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-06-30
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper introduces nf-core/mag to test whether a reproducible, best-practice Nextflow pipeline that optionally combines short and long reads and leverages sample-wise group information for co-assembly and co-abundance binning can improve metagenome assembly continuity and completeness relative to standard single-sample, short-read-only approaches.
- ★ nf-core/mag is a Nextflow/nf-core pipeline for hybrid metagenome assembly, binning and taxonomic classification of MAGs. resource
- ★ The pipeline can optionally combine short and long reads (hybrid assembly) to increase assembly continuity. method
- ★ The pipeline can use sample-wise group information to perform co-assembly and to compute co-abundances for genome binning. method
- ★ The pipeline follows nf-core best-practice guidelines, using containers (Docker/Singularity) and versioning to ensure reproducibility and portability across compute systems. method
- ★ Among existing workflow-manager-based metagenomics pipelines, only Muffin supports hybrid assembly and none support group-wise co-assembly, distinguishing nf-core/mag. finding
- ★ On simulated time-series metagenomic data, both hybrid assembly and group-wise co-assembly increase assembly size, N50 and the number of reconstructed MAGs compared to short-read-only, sample-wise assembly. finding
- The pipeline has already been successfully applied in published microbial studies. finding
- Future versions are planned to add functional annotation and assembly/bin refinement steps. method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| metagenomic data simulation | simulated microbial community (791 genomes, CAMI II challenge toy mouse gut dataset) | none (in silico simulation) | simulated Illumina and Nanopore sequencing reads across a time series | CAMISIM |
| de novo metagenome assembly | simulated time-series metagenomic samples (2 groups of 4 samples each) | assembly strategy: short-read-only vs hybrid; sample-wise vs group-wise co-assembly | assembly metrics: total length, N50, number of contigs, largest contig size | MEGAHIT/SPAdes/hybridSPAdes; summarized by QUAST |
| genome binning | assembled contigs from simulated time-series samples | group-wise vs sample-wise co-abundance-based binning | number of MAGs (bins) identified per assembly | MetaBAT2 |
| MAG quality assessment | MAGs recovered from simulated data | none | completeness and contamination estimates via single-copy orthologs | BUSCO |
| taxonomic classification of MAGs | MAGs recovered from simulated data | none | taxonomic assignment of each MAG | GTDB-Tk and CAT/BAT |
| read preprocessing and quality control | raw short (Illumina) and long (Nanopore) reads | none | adapter/quality-trimmed reads; host/PhiX and phage-lambda contamination removed; QC metrics | fastp, Bowtie2, FastQC (short reads); porechop, NanoLyse, Filtlong, NanoPlot (long reads) |
| taxonomic profiling of preprocessed reads | preprocessed short reads | none | community composition/contamination assessment prior to assembly | Kraken2 or Centrifuge, visualized with Krona |
- ▲ Hybrid assembly (short+long reads) increased total assembly size, N50 value, and number of reconstructed MAGs compared to short-read-only assembly.
- ▲ Group-wise co-assembly increased total assembly size, N50 value, and number of reconstructed MAGs compared to sample-wise assembly.
- – nf-core/mag uniquely supports group-wise co-assembly among compared workflow-manager-based pipelines (Muffin, ATLAS).
- – nf-core/mag was rated as providing reproducibility and adherence to strict best-practice development guidelines, unlike Muffin and ATLAS.
- count 791 genomes (genome sources used from the CAMI II toy mouse gut dataset for CAMISIM simulation)
- count 2 groups of 4 samples each (simulated time-series metagenomic dataset structure)
- other 4 assembly settings compared (short-read sample-wise, hybrid sample-wise, short-read co-assembly, hybrid co-assembly) (comparison of nf-core/mag assembly settings on simulated data)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a software application note introducing the nf-core/mag metagenome assembly and binning pipeline. The 'Results' section presents an illustrative demonstration on CAMISIM-simulated metagenomic time series data (2 groups × 4 time points). The comparison of four assembly settings (short-read vs. hybrid × sample-wise vs. group-wise co-assembly) was conducted entirely descriptively, reporting standard assembly quality metrics (total length, N50, contig count, largest contig, MAG count) visualised as scatter plots. No formal inferential statistical tests, p-values, or measures of dispersion were applied or reported; the authors explicitly caveat that the results apply to 'this particular time series data'.
-
The four assembly-setting conditions were compared by visually inspecting scatter plots of assembly metrics with no formal testing↳ Could also: A repeated-measures or paired design (e.g., Wilcoxon signed-rank or paired t-test across the eight samples treated as matched observations) could accompany the visual comparison — Because each sample passes through all four settings, the observations are naturally paired; a paired test would quantify the magnitude and consistency of differences across samples rather than relying solely on visual inspection
-
The illustrative experiment used a single simulated dataset (one realisation of the CAMISIM simulation)↳ Could also: Running multiple independent simulation replicates (different random seeds) and reporting summary statistics (mean ± SD) across replicates — A single realisation cannot distinguish systematic differences between settings from simulation stochasticity; multiple replicates would support more generalisable conclusions about which setting performs better on average
-
Assembly quality was summarised using absolute metric values (total length, N50, contig count) with no normalisation to the known reference↳ Could also: Reference-based evaluation metrics such as genome fraction covered, misassembly rate, or NGA50 (as provided by QUAST in reference mode, and used in CAMI benchmarks) could also be reported alongside — Since the ground-truth genomes are known (CAMISIM simulation from the CAMI II mouse gut dataset), reference-aware metrics more directly measure assembly correctness, complementing the reference-free metrics already shown
-
The number of recovered MAGs was reported as a raw count↳ Could also: Reporting MAG recovery in terms of completeness and contamination bins (e.g., fraction of high-quality MAGs per MIMAG standards: ≥90% completeness, ≤5% contamination) against the known reference genomes — Raw MAG count does not distinguish high-quality from fragmented or contaminated bins; quality-stratified recovery rates are widely used in CAMI-style benchmarks and would provide additional interpretive depth
-
The comparison was limited to one simulated community type (mouse gut metagenome)↳ Could also: Including a second community type with different complexity or abundance distribution (e.g., a lower-diversity mock community or a real public dataset) as an additional benchmark condition — Generalisability of the observed setting-dependent differences across community types cannot be inferred from a single community; a second community would illustrate the range of conditions under which each setting is advantageous
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — PMID 35118380 (nf-core/mag)
Paper: Krakau, Straub, Gourlé, Gabernet, Nahnsen. "nf-core/mag: a best-practice pipeline for metagenome hybrid assembly and binning." NAR Genomics & Bioinformatics 4(1):lqac007, 2022. DOI 10.1093/nargab/lqac007.
Nature of paper: A pipeline-description / application note. It introduces the nf-core/mag Nextflow pipeline and demonstrates it on simulated metagenomic data. The repo (https://github.com/nf-core/mag) is the authors' own pipeline — fully in scope per BRIEF rule P16 (third-party tool also valid, but here it is the authors').
Pipeline(s) named
- nf-core/mag (the subject of the paper), version v2.1.0 used for the demo. Tools inside: fastp/Porechop (QC), MEGAHIT + SPAdes/metaSPAdes/hybridSPAdes (assembly), QUAST (assembly metrics), MetaBAT2/MaxBin2 (+CONCOCT) (binning), BUSCO (bin completeness/contamination), GTDB-Tk + CAT (taxonomy), Prodigal/Prokka.
- CAMISIM — used to generate the simulated reads (data-generation step).
In scope (pipeline-derived, will attempt)
The central computational result is Figure 2 + its directional claim:
"The results demonstrate that both hybrid assembly as well as group-wise co-assembly increase the assembly's size, its N50 value and the number of reconstructed MAGs." (Results, ref. Fig. 2)
Figure 2 panels (assembly metrics across 4 nf-core/mag settings — sample-wise vs group-wise co-assembly × short-read-only vs hybrid):
- (A) Total assembly length
- (B) N50
- (C) Number of contigs
- (D) Largest contig size
- (E) Number of MAGs
Reproduction = run nf-core/mag on the paper's published Zenodo data (CAMISIM hybrid time-series; 2 groups × 4 timepoints; Illumina + Nanopore) in the relevant configurations and check the directional pattern of A/B/E (size↑, N50↑, MAGs↑ for hybrid vs short-read, and for group-wise vs sample-wise). The paper prints no absolute numbers in the main text (Fig 2 axes only); exact values live in Supplementary S4. So the gradeable target is the direction/ordering of the metrics, plus a functional "pipeline runs end-to-end on the published data and emits these exact metric files (QUAST assembly stats, BUSCO MAG counts)."
Secondary functional check: nf-core/mag -profile test reproduces (pipeline executes
and produces assembly + bins + QC reports) — validates the software artifact itself.
Out of scope (not pipeline-derived from the shipped data)
- Table 1 — qualitative Yes/No/Partial feature comparison of nf-core/mag vs Muffin vs ATLAS. Hand-curated feature matrix, not a computed result. Not attempted.
- CAMISIM simulation parameters (S4) — data generation; we consume the deposited reads rather than re-simulate them.
- Any wet-lab step: none (fully in-silico paper).
Reproduction strategy & feasibility
- Heavy compute → «our HPC» SLURM only («infra» work dir), per BRIEF.
- Full data ≈ 38 GB (8 hybrid samples). Full 4-way Figure 2 reproduction is a large
multi-day metagenomics run. Plan, in order of increasing cost:
-profile testend-to-end (software reproduces) — quick floor.- Real Zenodo data, one group's time series, short-read-only vs hybrid (sample-wise) → tests the core "hybrid ↑ size/N50/MAGs" claim on the paper's own data. Clear 1:1 data point.
- Add group-wise co-assembly and/or the second group → fuller Figure 2.
- Pin nf-core/mag v2.1.0 for faithfulness (paper's version); note current release differences if v2.1.0 is unrunnable with modern Nextflow.
Status note
«our HPC» VPN tunnel was DOWN at first contact (no compute submitted yet). Prep (scope/claims/dataset profile) done off-«our HPC»; compute resumes when tunnel returns.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.