Accurate prediction of metagenome-assembled genome completeness by MAGISTA, a random forest model built on alignment-free intra-bin statistics.
The main results reproduced: recomputed values matched the published ones within tolerance.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
REPRODUCED. MAGISTA (PMID 35248155) is the authors' own code (github LM-UGent/MAGISTA b06c160 v0.2) and ships the alignment-free intra-bin feature tables WITH ground-truth labels (training_dataset.csv 628 bins, test_dataset.csv 425 bins), so the headline RF completeness/purity RMSE/R2 numbers reproduce end-to-end via the authors' own train_MAGISTA/predict_MAGISTA WITHOUT re-deriving features from the 42 GB raw metagenome (ENA PRJEB43026, profiled). On «our HPC» (R 4.3.3, randomForest 4.7.1.2, «job»): the TITLE claim, MAGISTA completeness on the full test set, reproduced BIT-EXACT (RMSE 16.87 / R2 0.777, identical to the paper after flooring). 6 further metric pairs (CheckM baselines C2/C4, MAGISTA purity C3, real/sim completeness C5/C6, sim purity C7s) reproduced within tolerance. One partial: real-subset purity R2 (C7r) 0.307 vs reported 0.204 (RMSE matches; R2 fragile on that low-variance subset). MAGISTIC (C8) is BLOCKED — its code needs checkm.Marker.lineage, a column absent from both shipped CSVs (shipped-data gap, flagged for the auditor; reported 12.75/0.873 not derivable from shipped data). Two honest code/data findings recorded: (1) a typo (nfolds vs folds) in the authors' cv.RFpredict prevents the shipped code from running on the shipped (bin_group-less) data; we applied a numerically-inert 1-char fix; (2) paper states 842 training bins but the shipped CSV has 628 (headline still reproduces exactly from the 628). NOT attempted: de-novo GenDisCal feature extraction from raw reads; wet-lab assembly/binning.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-19 ⛓ db113eb8014b
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-25
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper tests whether an alignment-free, whole-bin (reference-independent) statistical approach to assessing metagenome-assembled genome (MAG) quality can overcome the accuracy and coverage limitations of existing single-copy marker gene (SCMG)-based tools like CheckM.
- ★ MAGISTA, a random forest model built on alignment-free intra-bin distance-distribution statistics, can estimate MAG completeness and purity without relying on reference marker genes. method
- ★ MAGISTA achieved a 20% reduction in root-mean-square error compared to the marker gene approach (CheckM) when tested on publicly available mock metagenomes. finding
- ★ A highly complex genomic DNA mock community of 227 bacterial strains spanning major phylogenetic lineages of cultivable bacteria was constructed as a benchmarking resource for metagenome analysis methods. resource
- ★ SCMG-based tools (e.g., CheckM) are limited because marker genes may be missing, of poor quality, or too taxonomically distant, and they only assess a small covered fraction of a MAG, potentially causing quality overestimation. finding
- GUNC, despite addressing some CheckM shortcomings, remains a gene-centric, reference-dependent approach. finding
- ★ Optimal fragment length and signature method (PaSiT4, MMZ3, MMZ4, Freq4) for distinguishing genomes were selected via QDA classifier accuracy on PCA-reduced signature distributions from 5-genome test sets. method
- 66 predictor variables (distance-distribution statistics, GC content distribution, bin fraction/comparisons, bin size) were derived per bin and used as random forest input. method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Shotgun metagenome sequencing (Illumina Novaseq 6000, 2x150bp PE) | Pooled genomic DNA mock community of 227 bacterial strains (HC227) | none | 60 Gb sequencing output used for assembly/binning training data | Illumina Novaseq 6000 |
| De novo assembly and binning (SPAdes + CONCOCT/MaxBin/MetaBAT2) | HC227 training mock metagenome and 5 public test mock metagenomes (BMock12, Rinke, MBARC-26, ZymoCS, Quince) | binning method/parameter variation (composition only vs composition+coverage) | metagenome-assembled genome bins | SPAdes 3.14; CONCOCT; MaxBin; MetaBAT2; Bowtie2 for coverage |
| Alignment-free pairwise distance computation (PaSiT4, MMZ3, MMZ4, Freq4) between contig fragments within bins | Contig fragments from HC227 and test mock metagenome bins | fragment length varied (1-100 kb) | distance distribution statistics (mean, SD, skewness, kurtosis, median, percentiles) per bin | R v4.0.3 |
| Quadratic discriminant analysis (QDA) classification on PCA-reduced signature distributions | 4 sets of 5 genomes from different phyla | none | classifier accuracy distinguishing two most-overlapping genomes, used to select fragment length/method | R MASS package (qda function) |
| Random forest regression modeling | 66 predictor variables (log-transformed, PCA-projected) from HC227 training bins | none | predicted bin completeness and purity | R RandomForest package (default parameters) |
| Gold-standard reference-based bin quality assessment (MetaQUAST + AMBER) | Mock metagenome bins with known reference genomes | none | bin completeness (matching bp/genome size), purity (matching bp/bin size), F1 score | MetaQUAST; AMBER |
| Marker-gene-based quality assessment (CheckM) | 5 public test mock metagenomes | none | completeness, contamination-derived purity | CheckM (43 SCMGs) |
- ▼ MAGISTA reduced root-mean-square error by 20% relative to the CheckM marker-gene approach on public mock metagenomes 20% reduction in RMSE
- ▲ Freq4 and MMZ3/MMZ4 signatures reached high QDA classification accuracy (up to ~0.97-0.99) at longer fragment lengths, guiding selection of method/fragment-length combinations accuracy 0.60-0.99 across methods/fragment sizes
- fold_change 20% reduction in RMSE (MAGISTA vs CheckM marker gene approach on public mock metagenomes)
- count 227 (bacterial strains pooled to build HC227 training mock community)
- count 66 (predictor variables used as random forest model input)
- other 0.60-0.99 (range of QDA classifier accuracies across PaSiT4/MMZ3/MMZ4/Freq4 methods and 1-100kb fragment lengths)
- count ~12,000 (number of sequenced bacterial type strain genomes)
- count 10^6-10^12 (estimated range of total microbial species diversity on Earth)
- count ~20,000 (validly named bacterial species, ~60% with a genome-sequenced type strain)
- count 43 (conserved single-copy marker genes (SCMGs) used by CheckM)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a computational tool-development study introducing MAGISTA, a random forest regression model that predicts metagenome-assembled genome (MAG) completeness, purity, and F1 score from 66 alignment-free, reference-independent predictor variables. The model was trained on bins derived from a 227-strain genomic DNA mock community and evaluated against CheckM on five external mock metagenome datasets using root-mean-square error (RMSE) and an R²y~x metric comparing predicted to observed bin quality. Fragment-length selection for the distance signatures was first optimized via quadratic discriminant analysis accuracy across four five-genome sets.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Quadratic discriminant analysis (QDA) | Optimization of fragment length and signature method combinations (Table 2); classifier trained to separate the two most overlapping genomes within each five-genome set | 4 five-genome sets (20 genomes total) | not stated |
| Random forest regression (default parameters) | Main MAGISTA model predicting bin completeness, purity, and F1 score from 66 predictor variables; trained on HC227 bins and tested on 5 external mock datasets | Bins from HC227 mock community (exact bin count not stated); 5 external test datasets (10–210 strains each) | not stated |
| Principal component analysis (PCA) | Dimensionality reduction of 66 log-transformed predictor variables for the training set; test sets projected onto training-derived PCs; also used during fragment-length optimization | — | na |
| Linear regression (lm) | Mentioned in Methods as part of the implemented procedure; specific application within the pipeline not further detailed in the available text | — | not stated |
| Root-mean-square error (RMSE) and R²y~x (Nash-Sutcliffe-style explained-variance coefficient) | Primary performance evaluation comparing MAGISTA vs. CheckM predictions against MetaQUAST+AMBER ground-truth bin quality across all test datasets | All bins across five external mock metagenome test datasets (exact n per dataset not stated in available text) | na |
-
The random forest model was trained with default hyperparameters (number of trees, mtry, minimum node size, etc.) without a reported tuning step↳ Could also: Systematic hyperparameter tuning via grid search or random search with cross-validation (e.g., using R packages caret or tidymodels) — Reporting a tuning procedure — even one that confirms defaults are adequate — makes modeling choices transparent and allows readers to judge whether performance could be further improved or is already near an optimum
-
Model performance was evaluated on five fixed external test datasets without cross-validation of the training procedure itself↳ Could also: k-fold cross-validation or leave-one-dataset-out cross-validation on the training set to generate internal performance estimates with associated variance — Cross-validated estimates convey the stability of the model and how sensitive performance is to the particular choice of training data; a single train/external-test split yields a point estimate without indicating variability
-
RMSE and R²y~x were reported as single point estimates with no uncertainty quantification around them↳ Could also: Bootstrap confidence intervals around RMSE or R²y~x, or stratifying reported metrics per individual test dataset to reveal dataset-to-dataset variability — Uncertainty bounds around performance metrics help readers assess whether the observed RMSE reduction over CheckM is consistent across datasets or driven by a subset, and whether differences exceed sampling variability
-
Fragment-length optimization used mean QDA classification accuracy across four five-genome sets as the sole selection criterion↳ Could also: Cross-validated classification accuracy, AUC-ROC, or comparison across multiple classifier types (e.g., linear SVM, k-NN) using the same four-set structure — Cross-validated accuracy reduces optimism; testing multiple classifiers would indicate whether the ranking of fragment lengths is classifier-agnostic rather than specific to QDA's assumptions
-
MAGISTA was quantitatively benchmarked against CheckM as the sole comparator method↳ Could also: Including BUSCO, GUNC, and/or Anvi'o (all introduced in the Background) in the quantitative RMSE/R² comparison table — A broader benchmark panel would situate MAGISTA within the full landscape of existing tools and help readers understand whether gains are specific to CheckM's marker-gene limitations or general across reference-based approaches
-
Bin quality was modeled as a continuous regression target (completeness, purity, F1 as 0–100% values)↳ Could also: Framing quality prediction as an ordinal or binary classification problem aligned with MIMAG quality tiers (high/medium/low quality), and evaluating with Matthews correlation coefficient or balanced accuracy — MIMAG categorical tiers are the practical standard researchers use when filtering MAGs for downstream analysis; classification metrics would directly quantify how well MAGISTA supports those real-world decisions
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-35248155 (MAGISTA)
Paper: Goussarov et al. 2022, Environ Microbiome 17:1. "Accurate prediction of metagenome-assembled genome completeness by MAGISTA, a random forest model built on alignment-free intra-bin statistics." DOI 10.1186/s40793-022-00403-7 · PMCID PMC8898458.
Repo: https://github.com/LM-UGent/MAGISTA — pinned commit
b06c16035709dfb8244c04fc6ded28b13c544167 (= tag v0.2, default branch main).
License GPL-2.0. This IS the authors' own code.
Data accession: ENA PRJEB43026 — the raw HC227 mock metagenome (1 run ERR5321934, NovaSeq 6000, paired, 573.6 M reads / 82.7 Gbp).
What MAGISTA does
A random forest (randomForest, ntree=200, maxnodes=60, set.seed(123456))
predicts MAG completeness and purity from alignment-free intra-bin
statistics: distributions (avg, sdev, skew, kurt, median, percentiles
2.5/5/10/90/95/97.5) of pairwise contig-fragment distances computed by GenDisCal
under several signatures — GC content (1 kb windows), tetranucleotide correlation
f4c (5 kb), Markov mid-order z m3c/m4c (10 kb), Karlin k4s (50 kb) — plus
bin size and bin-part fractions. Per target it builds rating-stratified submodels,
an mpm-PCA pre-transform, a 10-fold CV linear correction, then clamps to the
training value range. MAGISTIC = same + CheckM marker-gene columns.
In scope (pipeline-derived results we attempt)
The repo ships the derived feature tables with ground-truth labels, so the RF
prediction results are reproducible end-to-end from data/*.csv WITHOUT
re-deriving features from the 42 GB metagenome (the upstream GenDisCal feature
extraction is a separate, heavier pipeline — see "Attempted-light" below).
| id | reported result (paper) | pipeline | repro path |
|---|---|---|---|
| C1 | MAGISTA completeness on all test data: RMSE 16.87, R² 0.777 | RF train on training_dataset.csv → predict test_dataset.csv |
eval.R |
| C2 | CheckM completeness baseline: RMSE 20.05, R² 0.685 | checkm.Completeness vs actual |
eval.R |
| C3 | MAGISTA purity all test: RMSE 19.12, R² 0.365 | RF purity → predict | eval.R |
| C4 | CheckM purity baseline: RMSE 22.21, R² 0.143 | checkm.Purity vs actual |
eval.R |
| C5 | Real subset, completeness MAGISTA: R² 0.814, RMSE 14.73 | split by dataset (real = Bmock12/MBARC-26/Rinke/ZymoCS) |
eval.R |
| C6 | Simulated (Quince) completeness MAGISTA: R² 0.730, RMSE 18.81 | split by dataset |
eval.R |
| C7 | Real purity MAGISTA: R² 0.204, RMSE 13.10; Sim purity: R² 0.240, RMSE 23.76 | split | eval.R |
| C8 | MAGISTIC completeness R² 0.873, RMSE 12.75; purity R² 0.449, RMSE 17.80; F1 R² 0.834, RMSE 13.79 | RF + CheckM cols | eval.R (see blocker) |
R²/RMSE formula taken verbatim from the repo's own MAGISTA_error_plot /
table_rsquared(...,"model"): R² = 1 − Σ(actual−pred)² / Σ(actual−mean)² ,
RMSE = √(mean((actual−pred)²)). Paper truncates (floor) R² to 3 dp, RMSE to 2 dp.
Known blocker (recorded, honest)
MAGISTIC (C8) is likely NOT reproducible from the shipped data: the code
builds checkm.SpecificityRating from training_data$checkm.Marker.lineage, but
the column checkm.Marker.lineage is absent from both training_dataset.csv
and test_dataset.csv (they carry only checkm.Completeness/Contamination/
Purity/ngenomes/nmarkers/nmarker.sets/Strain.heterogeneity). Will attempt; if it
fails this is a docs_insufficient/shipped-data-gap note for the MAGISTIC rows.
Out of scope / attempted-light
- GenDisCal feature extraction from the 42 GB raw metagenome (the upstream
run_eval.shC-tool pipeline: nucops2 + genosig + GenDisCal). Heavy; the feature tables are already shipped, so re-deriving them is a separate verification we do NOT need for the headline RF numbers. Profiled only. - Wet-lab mock assembly / binning (CONCOCT, MaxBin, MetaBAT2) — out of scope.
- Cross-validation supplementary (Additional file 5) — sec
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.