Large-scale quality assessment of prokaryotic genomes with metashot/prok-quality.
The main results reproduced: recomputed values matched the published ones within tolerance.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No relevant deviation in data/preprocessing
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
REPRODUCED (1:1, software-tool paper, P16). metashot/prok-quality v1.2.0 (the exact paper version, bundled in Zenodo 4475355) was run on its own 8 deposited test genomes with default parameters via Nextflow 23.10.1 + Singularity/Apptainer on a «our HPC» compute node (SLURM 2247235, COMPLETED 14m25s, full CheckM tree). It emits the documented genome_info.tsv (all columns) with the stated tool versions (CheckM 1.1.2, GUNC 1.0.1, Barrnap 0.9, tRNAscan-SE 2.0.6, dRep 2.6.2); filtering drops exactly the low-completeness (other1) and GUNC-chimeric (other2) genomes; dRep collapses the 6 filtered to 3 representatives; GUNC flags the chimera. This is a tool paper with NO per-genome number table, so claims are graded on executability + output-schema + behavioural fidelity, all exact. Six SLURM iterations were needed to clear environment blockers (Nextflow config parser, jar 403, container TMPDIR, SIF-pull race, «infra» quota lag, tRNAscan shared-TMPDIR hang) — none of which reflect on the pipeline's scientific reproducibility. NOT attempted: any large-scale reference-collection benchmark (paper reports none); wet-lab steps (none exist).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 80assessed: 2026-06-19 ⛓ 415573f4f25a
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-25
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper addresses the problem that assessing the quality of metagenome-assembled genomes (MAGs) requires complex, hard-to-reproduce multi-step pipelines, and proposes that a container-based, automated Nextflow workflow (metashot/prok-quality) can provide reproducible, MIMAG-standard-compliant quality assessment and dereplication of draft prokaryotic genomes.
- ★ metashot/prok-quality is a container-enabled Nextflow pipeline for quality assessment and dereplication of draft prokaryotic genomes resource
- ★ The pipeline produces genome quality reports compliant with the MIMAG standard finding
- ★ The workflow runs out-of-the-box on any platform supporting Nextflow with Docker or Singularity, including clusters and cloud infrastructure method
- CheckM is used to estimate completeness, contamination, strain heterogeneity, and basic assembly statistics method
- ★ GUNC was added because SCG-based tools like CheckM have low sensitivity to non-redundant contamination from unrelated organisms; GUNC detects chimerism from both redundant and non-redundant contamination mechanism
- Barrnap predicts 5S/23S/16S rRNA genes and tRNAscan-SE predicts tRNA genes/isotypes using bacterial and archaeal models method
- dRep dereplicates filtered genomes by whole-genome ANI similarity, selecting cluster representatives via a CheckM-based quality score method
- The default workflow requires at least 70 GB RAM (imposed by CheckM), reducible to ~16 GB using --reduced_tree and --max_memory options finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| genome completeness/contamination/strain heterogeneity estimation | draft prokaryotic genomes (MAGs/bins from metagenome assembly and binning) | none | completeness, contamination, strain heterogeneity, basic assembly statistics (e.g. N50, genome size) | CheckM v1.1.2 |
| chimerism/non-redundant contamination detection | draft prokaryotic genome bins | none | lineage homogeneity of contigs, GUNC pass/fail | GUNC v1.0.1 |
| ribosomal RNA gene prediction | draft prokaryotic genomes | none | presence/absence of 5S, 23S, 16S rRNA genes | Barrnap v0.9 (HMM, bacteria and archaea databases) |
| transfer RNA gene prediction | draft prokaryotic genomes | none | number of tRNA genes and tRNA isotypes | tRNAscan-SE v2.0.6 (bacteria and archaea covariance models) |
| genome dereplication (ANI-based clustering) | filtered draft prokaryotic genomes (passing completeness/contamination/GUNC filters) | none | cluster membership, cluster representativeness, representative genome selection | dRep v2.6.2 |
| pipeline validation use case (reviewer testing) | chimeric bacterial genome vs. two corresponding clean genomes | none | contamination flag and inclusion/exclusion from downstream dereplication | metashot/prok-quality v1.2.0 pipeline |
- – The pipeline outputs a single comprehensive table (genome_info.tsv) with completeness, contamination, GUNC filter status, N50 and other assembly stats, rRNA gene presence, and tRNA counts, satisfying MIMAG reporting requirements
- – GUNC accurately detects chimerism induced by both redundant and non-redundant contamination, addressing a blind spot of SCG-based tools
- – dRep selects the cluster representative genome using the score formula: score = completeness − 5 × contamination + 0.5 × log(N50)
- – Default workflow requires at least 70 GB RAM (CheckM-imposed); can run on a 16 GB RAM workstation using --reduced_tree and --max_memory 16.GB 70 GB default vs. ~16 GB reduced
- – In independent reviewer testing, the pipeline with default thresholds flagged a known chimeric bacterial genome as contaminated and excluded it from dereplication, while the two corresponding clean genomes passed through all pipeline steps
- other at least 70 GB RAM required (default, imposed by CheckM v1.1.2) (resource requirement)
- other ~14 GB RAM with --reduced_tree, requires --max_memory 16.GB (reduced memory mode for workstation use)
- other default ANI threshold 0.95 for dereplication (must be > 0.90) (dRep clustering parameter)
- other default min_overlap 0.30 (minimum alignment overlap required to compute ANI)
- other default min_completeness 50%, max_contamination 10% (genome filtering thresholds)
- other score = completeness − 5 × contamination + 0.5 × log(N50) (dRep cluster representative selection formula)
- count 5 main modules (pipeline architecture (CheckM, GUNC, Barrnap, tRNAscan-SE, dRep))
- other tested on Nextflow v20.07.1 (software compatibility/testing)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a software tool article introducing metashot/prok-quality, a Nextflow-based containerised pipeline for quality assessment of metagenome-assembled genomes (MAGs). No inferential statistical tests are applied; the pipeline instead computes and reports deterministic genome quality metrics (completeness, contamination, assembly statistics, rRNA/tRNA gene presence) by orchestrating established bioinformatics tools. A fixed linear scoring formula is used for dereplication ranking, and genome clustering is performed via Average Nucleotide Identity (ANI) at a user-configurable threshold. The article contains no p-values, confidence intervals, or group comparisons.
-
Genome completeness and contamination are estimated using CheckM's lineage-specific single-copy core gene (SCG) catalogs↳ Could also: BUSCO (Benchmarking Universal Single-Copy Orthologs) or CheckM2 (a machine-learning-based successor) could also be used to estimate completeness and contamination — Different tools use different reference SCG sets and underlying models; reporting estimates from more than one tool can reveal cases where lineage placement uncertainty causes divergent completeness estimates, helping users gauge metric robustness
-
Chimerism and non-redundant contamination are assessed by GUNC alone↳ Could also: Complementary manual or semi-automated inspection tools such as Anvi'o's genome refinement interface, or tetranucleotide-frequency visualisation, could also be applied — GUNC and SCG-based tools detect different contamination signatures; a multi-tool approach can increase sensitivity for chimeric bins that escape any single detection method
-
The dereplication scoring formula applies fixed weights: score = completeness − 5 × contamination + 0.5 × log(N50)↳ Could also: User-adjustable weighting coefficients, or alternative scoring schemes that also incorporate GUNC scores or strain heterogeneity, could also be used to rank cluster representatives — Optimal trade-offs between completeness, contamination, and assembly contiguity vary by downstream application; exposing the weights as parameters would let users tune representative selection to their specific needs
-
Dereplication clustering uses a single fixed ANI threshold (default 0.95) with a fixed minimum alignment overlap (0.30)↳ Could also: A two-step clustering approach (e.g., coarse Mash-distance pre-clustering followed by precise FastANI comparison at multiple thresholds) could also be applied — ANI 0.95 approximates species-level boundaries for many prokaryotes, but strain- or genus-level questions require different thresholds; a two-step approach can reduce compute cost at scale while preserving flexibility
-
rRNA gene prediction uses Barrnap with HMM models for bacteria and archaea separately↳ Could also: Infernal with the Rfam covariance model database, or RNAmmer, could also be used for rRNA prediction — Different predictors vary in sensitivity for divergent or novel lineages; noting the tool's known false-negative rate for highly divergent rRNA sequences would help users interpret absent-rRNA results in novel MAGs
-
Quality filtering applies fixed default thresholds (completeness ≥ 50 %, contamination ≤ 10 %) aligned with the MIMAG medium-quality standard↳ Could also: Reporting the full distribution of completeness and contamination values across all bins, rather than a binary pass/fail table, could also accompany the filtered output — A distribution summary (e.g., histogram or percentile table) would allow users to see how many bins fall just below thresholds and to calibrate cut-offs to their specific study requirements without re-running the pipeline
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-35136576
Paper: Albanese D, Donati C. Large-scale quality assessment of prokaryotic genomes with metashot/prok-quality. F1000Research 2021, 10:822. DOI 10.12688/f1000research.54418.1 · PMID 35136576 · PMCID PMC8804904
Code: https://github.com/metashot/prok-quality
Data: Zenodo 10.5281/zenodo.4475355 ("metashot/prok-quality v1.2.0 with test data",
single archive prok-quality-1.2.0_with_test_data.zip, 7.26 MB, MD5 181310e9e676867004aee9ce55a61520)
Nature of the publication
This is a software / methods (tool) paper published as an F1000Research
"Software Tool Article". It describes a Nextflow pipeline (metashot/prok-quality)
that wraps existing third-party tools to assess the quality of prokaryotic genomes /
MAGs (metagenome-assembled genomes) against the MIMAG/MISAG standard. It is a
P16-type reproduction unit (running an existing/own tool on its own deposited
test data is equally valid per BRIEF rule 2). The paper reports essentially no
large empirical result table — there is no benchmark dataset with reported
completeness/contamination numbers per genome. Its verifiable, pipeline-derived
content is therefore: (a) the pipeline runs end-to-end on the deposited test
data and emits the documented outputs, and (b) the documented parameters,
tool versions, filtering logic and scoring formula are exactly as the pipeline
implements them.
Version pinning (paper era)
Paper (June 2021) reports tool versions: CheckM 1.1.2, GUNC 1.0.1,
Barrnap 0.9, tRNAscan-SE 2.0.6, dRep 2.6.2.
The repo tag whose process.config containers match these EXACTLY is
1.2.3 (commit 7873ba8e746f114e40f88e3835e491033ff0afc3, 2021-06-16):
metashot/checkm:1.1.2-2, metashot/gunc:1.0.1-2, metashot/barrnap:0.9-1,
metashot/trnascan-se:2.0.6-2, metashot/drep:2.6.2-2.
(Repo master has drifted to GUNC 1.0.5 — NOT used.) The Zenodo deposit bundles
v1.2.0; v1.2.3 is the same minor line with identical tool versions, chosen as the
run pin. Drift between 1.2.0 and 1.2.3 is documented in AUDIT.md.
IN SCOPE (pipeline-derived → attempted)
| id | result | source in paper | how reproduced | type |
|---|---|---|---|---|
| R1 | Pipeline runs end-to-end on the deposited test genomes and produces genome_info.tsv with the 12 documented columns (Genome, Completeness, Contamination, Strain heterogeneity, GUNC pass, Genome size (bp), # predicted genes, 5S rRNA, 23S rRNA, 16S rRNA, # tRNA, # tRNA types) |
Fig./README output description | run prok-quality 1.2.3 on Zenodo test data on «our HPC» SLURM | empirical (executability + output schema) |
| R2 | Tool versions CheckM 1.1.2 / GUNC 1.0.1 / Barrnap 0.9 / tRNAscan-SE 2.0.6 / dRep 2.6.2 | Methods / "Operation" | inspect pinned process.config containers + record runtime --version of each tool in env |
code inspection + runtime |
| R3 | Default parameters: min_completeness=50, max_contamination=10, ani_thr=0.95, min_overlap=0.30, max_memory≈70 GB |
"Use" / parameter table | inspect pinned nextflow.config/process.config |
code inspection (exact) |
| R4 | Filtering rule: genome passes iff Completeness ≥ min_completeness AND Contamination ≤ max_contamination (+ optional GUNC pass) | "Use" | inspect bin/genome_filter.py + verify on run outputs |
code inspection + empirical |
| R5 | MIMAG high-quality draft = completeness >90 % AND contamination <5 % AND 5S+16S+23S rRNA present AND ≥18 tRNA types; medium = completeness ≥50 % AND contamination <10 % | "Introduction"/README | apply rule to the per-genome genome_info.tsv produced (functional sanity: a complete reference genome → HQ) |
empirical (functional) |
| R6 | dRep dereplication score = completeness − 5×contamination + 0.5×log(N50) | README/Methods | dRep 2.6.2 default scoring; verify against dRep behaviour | behavioural |
| R7 | Memory: ≥70 GB RAM required; ~14–16 GB with --reduced_tree |
"Operation" | config max_memory default |
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.