binny: an automated binning algorithm to recover high-quality genomes from complex metagenomic datasets.
The main results reproduced, with only marginal, non-material deviations.
- ✓The central claim held under reproduction
- 🟡Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
PARTIAL with a strong, healthy core; no fabrication signal. PRIMARY deterministic target C1 REPRODUCED EXACTLY: ran a-h-b/binny v2.2.15 (confirmed manuscript version; brief's code URL nathanhaigh/3521724 is dead 404, used canonical tool per P16) end-to-end on «our HPC» SLURM and got exactly the documented 4 bins (full Snakemake/conda/Prokka/Mantis/CheckM/HDBSCAN stack, 9/9 steps). Required several packaging-compat fixes (mamba 2.8.1->1.5.8, tabulate 0.9->0.8.10, NLTK data, hdbscan cachedir->location, short TMPDIR for AF_UNIX sockets) all of which are result-preserving. Table 1 = MIMAG MAGs (purity>95% & completeness>90% AND >=18 tRNA & >=3 rRNA). T2 re-derivation from Zenodo 6977322: real-world HQ count C13=3013 re-derives EXACTLY; dataset N exact (54 CAMI samples, 105 real-world metagenomes). For every MIMAG claim (C2-C9) the recomputable completeness/purity component (NC) is consistently >= the reported MIMAG count, exactly as expected for a stricter subset -- but the exact MIMAG numbers are NOT independently re-derivable because the deposit lacks the per-bin tRNA/rRNA annotations. Per-binner cross-check (CAMI High): SemiBin slightly leads at NC level (360 vs 337) but binny leads after the MIMAG tRNA/rRNA filter (164 vs 148), corroborating the paper's stated reason for binny's advantage. NOT attempted: full 54-sample+105-metagenome benchmark re-run and competing binners (out of scope); T3 full-CAMI binny run (needs external CAMI assemblies).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 67assessed: 2026-06-22 ⛓ 2c1e7a990570
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-22
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-22no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper tests whether an automated binning algorithm (binny) using iterative, nonlinear dimension reduction of k-mer composition and read coverage combined with lineage-specific marker gene set assessment can recover more high-quality, high-purity metagenome-assembled genomes—including from highly fragmented assemblies—than existing state-of-the-art binning methods.
- ★ binny outperforms or is highly competitive with commonly used and state-of-the-art binning methods (MetaBAT2, MaxBin2, CONCOCT, VAMB, SemiBin, MetaDecoder) finding
- ★ binny recovers unique genomes that could not be detected by other binning methods finding
- ★ binny almost always recovers the most near-complete (>95% pure, >90% complete) and high-quality (>90% pure, >70% complete) genomes from CAMI simulated datasets compared to other tested methods finding
- ★ binny returns substantially more high-quality (MIMAG-standard) draft genomes than any other tested method from a real-world benchmark spanning multiple ecosystems finding
- ★ binny improves combined MAG refinement results when used as input to MetaWRAP and DAS Tool finding
- ★ binny uses iterative, nonlinear dimension reduction (openTSNE) of k-mer frequency and coverage signatures followed by HDBSCAN clustering and CheckM-based marker gene set assessment to bin contigs method
- binny is implemented as a reproducible Python-based Snakemake workflow resource
- Single contigs with at least 40 distinct markers, ≥90% purity and ≥92.5% completeness are extracted directly as single-contig MAGs before iterative binning method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| genome binning benchmark | CAMI (Critical Assessment of Metagenome Interpretation) simulated metagenomic datasets (six datasets) | none (algorithm comparison) | MAG completeness and purity relative to reference genomes; number of near-complete/high-quality MAGs recovered | — |
| genome binning benchmark | real-world metagenomic datasets from a wide range of ecosystems | none (algorithm comparison) | number of MIMAG-standard high-quality draft genomes recovered | — |
| binning tool comparison | metagenomic contigs (simulated and real-world) | none | completeness, purity/contamination of resulting MAGs | MetaBAT2, MaxBin2, CONCOCT, VAMB, SemiBin, MetaDecoder |
| MAG refinement evaluation | binned contig sets from binny and other binners | combination/refinement of multiple binners' output | improvement in overall yield and quality of MAGs | MetaWRAP, DAS Tool |
| gene calling and RNA/CRISPR annotation | assembled metagenomic contigs | none | ORF predictions, rRNA/tRNA gene predictions, CRISPR region predictions | Prokka (using Prodigal, barrnap, minced) |
| marker gene set annotation (HMM search) | predicted coding sequences from metagenomic contigs | none | presence/counts of lineage-specific single-copy marker genes for completeness/purity assessment | Mantis, using CheckM PFAM and TIGRFAM HMM sets |
| contig depth of coverage calculation | metagenomic sequencing reads mapped to contigs | none | average per-contig read coverage depth | BEDTools genomeCoverageBed |
| dimensionality reduction and clustering | metagenomic contig k-mer and coverage feature matrices | none | two-dimensional embedding coordinates and contig clusters | openTSNE (with PCA pre-reduction), HDBSCAN |
- ▲ binny recovers the most near-complete (>95% pure, >90% complete) and high-quality (>90% pure, >70% complete) genomes from CAMI simulated datasets compared to other tested binning methods
- ▲ binny returns substantially more MIMAG-standard high-quality draft genomes than any other tested method from real-world ecosystem metagenomes
- ▲ binny contributes uniquely identified MAGs not found by other binning algorithms
- ▲ Incorporating binny improves combined refinement results with MetaWRAP and DAS Tool
- other >95% purity, >90% completeness (definition of 'near-complete' MAG quality category used for evaluation)
- other >90% purity, >70% completeness (definition of 'high-quality' MAG quality category used for evaluation)
- other ≥40 markers, ≥90% pure, ≥92.5% complete (thresholds for extracting single-contig genomes directly as MAGs before iterative binning)
- other default minimum contig length 500 bp (default short-sequence filtering threshold in binny)
- other initial completeness threshold 92.5%, decreasing to minimum 72.5%; purity threshold 95% (or 92.5%/99% depending on marker set/completeness) (default cluster acceptance thresholds during iterative clustering)
- other default minimum completeness threshold 70% (final/minimum completeness threshold for co-assembly iterative binning routine)
- other k = 2, 3, 4 (default k-mer sizes used for binning features)
- other HDBSCAN min_samples default 1, 5, 10 (n=3 iterations) (default iterative clustering parameter)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper describes a computational benchmarking study of a metagenomic binning algorithm (binny) against seven other binning tools, using simulated (CAMI) and real-world metagenomic datasets. Performance is assessed using established genome-quality metrics (completeness and purity/contamination based on CheckM marker gene sets, and MIMAG high-quality draft genome criteria) rather than classical inferential statistics, and results are reported as counts/proportions of recovered MAGs meeting quality thresholds across methods and datasets. The provided text (methods section) does not describe formal hypothesis testing, p-values, or variance/dispersion reporting for the benchmarking comparisons.
-
Binning tools are compared primarily by counting/tallying the number of near-complete, high-quality, or unique MAGs each method recovers across benchmark datasets.↳ Could also: A formal paired statistical comparison (e.g., Wilcoxon signed-rank test or paired t-test across datasets/genomes) or a mixed-effects model with dataset as a random effect could also be used. — This would let readers gauge whether the observed differences in recovery counts across methods and datasets exceed what could arise from dataset-to-dataset variability, complementing the descriptive tallies.
-
Genome quality is summarized using fixed completeness/purity percentage thresholds (e.g., >95% pure, >90% complete) to classify MAGs as near-complete or high-quality.↳ Could also: Reporting the full distribution of completeness and purity values (e.g., via boxplots, medians with interquartile ranges, or density plots) for each method could also be used. — This would convey the spread and shape of quality scores across all recovered genomes, in addition to the proportion crossing a categorical threshold.
-
No confidence intervals or uncertainty measures accompany the reported counts of recovered MAGs per method.↳ Could also: Bootstrap resampling across contigs or datasets to generate confidence intervals around MAG recovery counts could also be used. — This would provide a sense of the stability of the ranking between binning methods given sampling variability inherent in metagenomic assembly and simulation.
-
Performance across CAMI simulated datasets and real-world datasets from different ecosystems is reported dataset-by-dataset without apparent pooling or an overall summary statistic.↳ Could also: A meta-analytic or mixed-model summary combining results across datasets, weighting by dataset size or complexity, could also be used. — This would yield a single overall estimate (with uncertainty) of relative method performance while still allowing dataset-specific results to be inspected.
-
Dimensionality reduction (t-SNE via openTSNE) and clustering (HDBSCAN) parameters were tuned using heuristics and prior literature-based defaults rather than through a systematic statistical optimization procedure described for the comparison itself.↳ Could also: A systematic hyperparameter sensitivity analysis (e.g., grid search with cross-validation on held-out simulated genomes) could also be used. — This would quantify how sensitive the binning results are to specific parameter choices (e.g., perplexity range, minimum cluster size) across the benchmark datasets.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-36239393 (binny)
Paper: Hickl O, Queirós P, Wilmes P, May P, Heintz-Buschart A. binny: an automated binning algorithm to recover high-quality genomes from complex metagenomic datasets. Brief Bioinform 2022. DOI 10.1093/bib/bbac431 · PMCID PMC9677464.
What binny is
A Snakemake-based metagenomic binning workflow. Inputs: assembled contigs (FASTA) + read alignments (BAM) or a contig depth file. It clusters contigs into metagenome-assembled genomes (MAGs) using composition (k-mer) + coverage + CheckM marker genes, with iterative cluster refinement (Prokka gene calls, Mantis marker annotation, CheckM completeness/purity, t-/rRNA detection for MIMAG).
Code & data sources (P16 note)
- Brief's code URL
github.com/nathanhaigh/3521724is DEAD (HTTP 404). - Canonical tool repo = github.com/a-h-b/binny (the paper's "Code availability"). Per HARD RULE P16, running the canonical/authoritative tool on the paper's data is an equally valid reproduction. We use a-h-b/binny.
- Manuscript scripts: github.com/ohickl/binny_manuscript.
- Deposited benchmark data/results: Zenodo 10.5281/zenodo.6977322 (2.5 GB) — AMBER assessment outputs (binny + competitors), input contig-depth files, real-world tables. It does NOT contain the raw CAMI assemblies/BAMs (those live in CAMI repositories).
Benchmark design (from Methods)
- 6 CAMI synthetic datasets: CAMI 1 High Complexity (5 samples) + CAMI 2 toy human microbiome Airways/Gastrointestinal/Oral/Skin/Urogenital — 54 samples total.
- Real-world: 105 metagenomes (from the MetaBAT2 publication).
- Competing binners: MetaDecoder, SemiBin, MetaBAT2, MaxBin2, VAMB, CONCOCT (+ DAS Tool, refiners). Assessment via AMBER (recall/purity/ARI) and MIMAG quality counts.
- Quality defs: NC (near-complete) ≥95% pure & ≥90% complete; HQ ≥90% pure & ≥70% complete (CheckM); MIMAG adds ≥18 unique tRNAs + ≥3 unique rRNAs.
IN SCOPE (pipeline-derived, attempted)
| Tier | Target | Pipeline | Feasibility |
|---|---|---|---|
| T1 | binny test run → exactly 4 bins (repo docs, config.test.yaml) |
full binny Snakemake on shipped test data | PRIMARY — deterministic, documented expected output; proves the whole pipeline (Snakemake/conda/CheckM/Prokka/Mantis) runs |
| T2 | Table 1 MIMAG MAG counts for binny (164/192/202/243/215/152 CAMI; 629 real-world) | re-derive from deposited AMBER per-genome results (Zenodo) | re-derivation / fabrication-check — verify the paper numbers are recomputable from the deposit |
| T3 (stretch) | binny HQ/NC MAGs on ONE CAMI sample | full binny on one CAMI assembly + deposited depth | heavy (Prokka/Mantis/CheckM per sample); attempt if T1/T2 land |
OUT OF SCOPE (not attempted, stated)
- Full 54-sample + 105-metagenome benchmark re-run (assembly + mapping + binning + CheckM for every sample) — computationally enormous, not feasible here.
- Re-running the competing binners (MetaDecoder/SemiBin/VAMB/...) — out of scope; we only reproduce binny's own outputs and (T2) cross-check the deposited comparison.
- The CAMI assemblies/mappings themselves (external, multi-TB upstream).
HARD constraints
- All heavy compute on «our HPC» SLURM; all data on «infra»; «host» = small results only.
- 2026-06-19: «our HPC» VPN tunnel was DOWN at start (cisco "Verbindung getrennt") — prep done locally; execution waits for central VPN fix (HARD RULE 1d: never touch VPN).
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.