Reproducible, portable, and efficient ancient genome reconstruction with nf-core/eager.
The main results reproduced: recomputed values matched the published ones within tolerance.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No authors-side cause for any deviation
- ✓Any deviation was negligible
- 🟡A deviation arose in the data or preprocessing
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
REPRODUCED (1:1). Software/pipeline paper (nf-core/eager). Re-ran the exact documented pipeline (nf-core/eager v2.2.0dev @ commit 830c22d, Nextflow 22.10.6, conda profile with tool versions identical to the paper's singularity image: bwa 0.7.17, AdapterRemoval 2.3.1, Picard 2.22.9, samtools 1.9, DamageProfiler 0.4.9, QualiMap 2.2.2d, MultiQC 1.9) on the Viking-age cod benchmark (ENA PRJEB20524: COD076=ERR1943600-02, COD092=ERR1943607-09) mapped to gadMor3.0 GCF_902167405.1, on «our HPC» via SLURM («job» COMPLETED rc=0, 53m) with data on «infra». All 18 Table 3 / supplement mapping metrics graded: 12 EXACT (processed reads, % endogenous, duplication, mean read length, coverage, 5' C>T damage — all to 4+ sig figs), 6 within-tolerance (bwa-aln stochastic read counts, all within the <=~2 sd run-to-run variance the paper itself documents across its 10 replicate runs), 0 mismatch. Reported ground-truth recovered from the paper supplement (apeltzer/eager2-paper, 10-replicate MultiQC CSVs). Exact match of all deterministic values is strong evidence the table is genuine pipeline output, no fabrication indicators. Every reproduced value traces to reproduction/outputs/*.txt. NOT attempted (out of scope): Table 2 runtimes (hardware-specific), EAGER v1/PALEOMIX comparison, Barquera human-aDNA metagenomic demonstration. All grades provisional (must_be_independently_checked=true).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-18 ⛓ 5c05b8025f49
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-25
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper does not test a scientific hypothesis but presents a redesigned bioinformatics tool, asking whether a completely re-implemented, Nextflow-based version of the EAGER pipeline can improve accessibility/adaptability, reproducibility, and adherence to current best practices for ancient genomic data analysis compared to existing pipelines (EAGER, PALEOMIX).
- ★ nf-core/eager is a complete redesign and extension of the EAGER pipeline in Nextflow, built within the nf-core framework to ensure high-quality, sustainable software development. resource
- ★ nf-core/eager supports automated processing and merging of complex multi-library, multi-lane, multi-sequencing-run data via an optional TSV input table, unlike the original EAGER which handled only a single library at a time. method
- ★ nf-core/eager adds ancient metagenomic screening functionality (MALT, Kraken2, HOPS/MaltExtract, BBduk) for pathogen and microbiome detection in off-target reads, extending beyond the original EAGER pipeline. method
- ★ nf-core/eager aggregates outputs from every supported tool into a single comprehensive MultiQC report, replacing the need to inspect many independent output files as required in EAGER/PALEOMIX. method
- ★ nf-core/eager provides native HPC scheduler and cloud computing integration plus software container/environment support (Conda, Docker, Singularity), features absent or partial in EAGER and PALEOMIX. finding
- ★ New analytical modules added versus original EAGER include fastp (poly-G artefact removal), FreeBayes and pileupCaller (variant/genotype calling), Sex.DetERRmine (biological sex determination), and ANGSD (nuclear contamination estimation). method
- A 'FASTQ host removal' script allows safe public sharing of metagenomic sequencing data by stripping reads mapped to a (human) reference, protecting personal data privacy. method
- mapDamage2 and Schmutzi were not carried over from EAGER because DamageProfiler is a faster successor and a stable Schmutzi release was unavailable on Bioconda at time of writing. method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Whole-genome sequencing / host genomic analysis and metagenomic pathogen screening (re-analysis, functionality demonstration) | Human skeletal remains (three 16th century individuals from a mass burial site, Mexico City) | none (re-analysis of previously published data from Barquera et al., 2020) | Genetic/genomic results and putative pathogen detection | — |
- count more than 1 billion reads (Sequencing reads required for 1X depth coverage of a Yersinia pestis genome due to low endogenous DNA content, as reported in Rasmussen et al., 2015)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a bioinformatics software paper describing nf-core/eager, a Nextflow-based pipeline for ancient genomic data processing. The paper does not report inferential statistics or hypothesis tests; instead it presents a qualitative feature-comparison table across three pipelines and a benchmarking/demonstration re-analysis of a previously published dataset. Quantitative outputs are computational performance and pipeline quality-control metrics rather than biological effect estimates.
-
Pipeline feature comparison across EAGER, PALEOMIX, and nf-core/eager was reported as a binary Yes/No/Partial table (Table 1) without quantitative performance metrics in the feature-comparison section↳ Could also: A structured benchmark with quantitative runtime, memory consumption, and throughput metrics (e.g., wall-clock time per sample, peak RAM) across the three pipelines on identical test datasets could also be reported — Quantitative benchmarks allow readers to estimate resource costs on their own hardware and make empirically grounded tool choices; binary feature tables convey capability breadth but not efficiency trade-offs
-
Pipeline reproducibility is demonstrated by re-running a published dataset (Barquera et al. 2020) and implicitly comparing outputs, but no formal concordance or agreement statistics are reported in the available text↳ Could also: Concordance metrics such as genotype concordance rates, Pearson or Spearman correlation of coverage estimates, or Bland–Altman analysis between pipeline outputs could also be reported — Formal agreement statistics quantify the degree to which the new pipeline reproduces prior results, providing a concrete reproducibility claim rather than a qualitative assertion
-
Software dependency management relies on Conda, Docker, and Singularity containers, with tool versions drawn from Bioconda at time of release; specific tool versions are not tabulated in the paper body↳ Could also: A versioned dependency table or a machine-readable software bill of materials (e.g., SBOM or Conda lock file excerpt) could also be included as a supplementary resource — Pinned version tables allow exact replication of the computational environment years after publication, which is particularly important for pipelines where upstream tool behaviour can change between releases
-
Ancient DNA damage and authenticity metrics (e.g., C-to-T substitution rates from DamageProfiler) are reported as pipeline outputs for the demonstration dataset but their uncertainty is not quantified in the available text↳ Could also: Bootstrap confidence intervals or Bayesian credible intervals on damage-rate estimates could also be reported, as implemented in tools such as mapDamage2's Bayesian mode — Interval estimates on damage rates allow users to distinguish genuine aDNA signal from sampling noise, particularly for low-coverage libraries where point estimates are highly variable
-
Metagenomic profiling results from MALT and Kraken2 are presented as pipeline outputs without a cross-tool concordance assessment in the available text↳ Could also: A sensitivity/specificity or F1-score comparison between the two profilers on a mock community or known-composition sample could also be reported — Tool-concordance data on a reference community helps users select or combine profilers appropriately for their target organisms and coverage levels
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-33777521 (nf-core/eager paper, Fellows Yates et al. 2021, PeerJ)
Paper: "Reproducible, portable, and efficient ancient genome reconstruction with nf-core/eager." PMID 33777521 · PMC7977378 · DOI 10.7717/peerj.10947.
This is a software/pipeline paper. The code artifact is the Nextflow pipeline
nf-core/eager (NOT stschiff/sequenceTools, which is only one bundled tool).
Benchmarking version pinned in the paper: v2.2.0dev, commit 830c22d.
In-scope (pipeline-derived, reproducible)
Table 3 — mapping statistics for the Viking-age cod (Gadus morhua) benchmark, produced by running nf-core/eager on two samples (3 Illumina PE runs each) from Star et al. 2017 (ENA PRJEB20524), mapped to gadMor3.0 (GCF_902167405.1). Reproduced by re-running the exact documented pipeline command:
nextflow run nf-core/eager -r dev --input <tsv> -c pub_eager_vikingfish.conf \
-profile pub_eager_vikingfish,singularity \
--fasta GCF_902167405.1_gadMor3.0_genomic.fasta \
--skip_fastqc --skip_preseq --run_bam_filtering \
--bam_mapping_quality_threshold 25 --bam_discard_unmapped \
--bam_unmapped_type 'discard' --dedupper 'markduplicates'
All other params = eager v2.2.0 defaults (mapper bwaaln, n=0.04 k=2 l=1024; AdapterRemoval collapse, clip_readlength 30, q20; Picard MarkDuplicates; DamageProfiler; QualiMap; endorSpy). The paper ran this 10× to get run-to-run variance; the per-replicate MultiQC general-stats CSVs are in the paper supplement (github.com/apeltzer/eager2-paper, content/supplement/benchmarking/results/) and are used here as the ground-truth reported values (see original/reported_values_table3.tsv).
Target metrics (per sample COD076, COD092): processed reads (post-AdapterRemoval), mapped reads (pre-filter), MAPQ>=25 filtered mapped reads, % endogenous DNA, % duplication, mean read length, post-dedup mapped reads, mean coverage, 5' C>T damage.
Input data (ENA PRJEB20524, Gadus morhua, HiSeq 2500, PAIRED)
- COD076 (lib COD076E1bL1): ERR1943600 (L8), ERR1943601 (L6), ERR1943602 (L1)
- COD092 (lib COD092E1bL1i69): ERR1943607 (L6), ERR1943608 (L7), ERR1943609 (L8)
- Reference: GCF_902167405.1_gadMor3.0_genomic.fna (RefSeq, NCBI FTP)
Out of scope (not pipeline-deterministic / not attempted)
- Table 2 runtimes (105.6 / 209.2 min etc.) — hardware-specific (GWDG cloud, 32 cores), NOT reproducible on different hardware. Recorded but not graded.
- EAGER v1 and PALEOMIX comparison numbers — require those other pipelines; tangential to reproducing nf-core/eager's own output.
- Barquera et al. 2020 human-aDNA demonstration (sex determination, pathogen screening with the full Oct-2017 NCBI nt database, zenodo 4382153) — extremely heavy (human genome mapping + metagenomic screen against full nt); deferred / likely not attempted. Qualitative claims (all male; T. pallidum + HBV detected) are not single reproducible numbers.
Execution
All compute on «our HPC» («infra») via SLURM, data on «infra». nf-core/eager run with the pinned conda environment.yml (same pinned tool versions as the singularity image: bwa 0.7.17, AdapterRemoval 2.3.1, picard 2.22.9, samtools 1.9, damageprofiler 0.4.9, qualimap 2.2.2d, multiqc 1.9). Singularity unavailable on «our HPC» without root → conda profile with identical pinned versions.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.