Prediction of Antimicrobial Resistance in Gram-Negative Bacteria From Whole-Genome Sequencing Data.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No relevant deviation in data/preprocessing
- ✓Any deviation was negligible
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough to reproduce the INFERENCE pipeline 1:1. wgs2amr (authors' own tool, commit 561f6d5) ships the trained XGBoost median models + a DIAMOND ARG database + example reads with a documented expected output. Ran the full shipped pipeline (DIAMOND blastx -> R feature build -> XGBoost prediction) on the shipped testFile.fastq.gz on «our HPC»; all 8 antibiotic predictions (call AND reliability) match the README's documented output EXACTLY (8/8 exact). The paired-end shipped reads gave a byte-identical table. Two undocumented legacy version pins were REQUIRED: diamond=0.8.36 (the 2020 .dmnd is rejected by diamond 0.9.x/2.2) and r-xgboost=0.90 (the shipped medianModels.rds is a pre-1.0 booster that xgboost>=2.x cannot load) - with default modern toolchains the pipeline fails. NOT attempted (out of scope): Table 3 performance metrics and Fig 3 AUC distribution, because the repo ships only the trained models + inference code, NOT the training pipeline (915-sample data prep, under-sampling, 51 random 70/30 splits, 10-fold CV, model selection); also USEARCH ARG clustering and the in-house WGS+phenotypic AST (wet-lab). PRJNA587095 (19 in-house isolates, SRR10383448-66) resolved but not graded - the paper prints no per-sample predictions to compare against. Verdict: the directly-reproducible inference surface reproduces exactly; the training-derived numbers were not reproducible from the shipped artifacts and are recorded as not-attempted, not as mismatch.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 100assessed: 2026-06-17 ⛓ 230630363a53
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-17
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-18no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusThe authors test whether whole-genome sequencing data, quantified as coverage of known antibiotic resistance genes, can be used with machine learning to accurately predict antimicrobial resistance to eight antibiotics across five Gram-negative bacterial species, despite the more complex genotype-to-phenotype relationship in Gram-negatives.
- ★ WGS data enables prediction of antimicrobial resistance in Gram-negative bacteria using machine learning finding
- ★ Eight XGBoost-based binary models, one per antibiotic, predict resistance to cefepime, cefotaxime, ceftriaxone, ciprofloxacin, gentamicin, levofloxacin, meropenem, and tobramycin method
- ★ Models are species-independent, trained without stratifying by bacterial species method
- ★ Models demonstrate high performance and robustness to class-imbalanced datasets, evaluated with the more stringent Matthew's correlation coefficient (MCC) finding
- ★ A reliability index (RI), the inverse of a misclassification rate, accompanies each prediction to assess prediction certainty method
- ★ Antibiotic resistance genes are clustered into ARGCs at >=90% identity, reducing input from 4579 ARG to 1027 ARGC (152 found in data) method
- ★ A publicly available tool (wgs2amr) provides an in silico antibiogram for eight drugs resource
- WGS-based prediction can inform clinical decisions within a day versus 2-4 days for conventional antibiograms finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Whole-genome shotgun sequencing (public data reuse) | A. baumannii, E. coli, E. cloacae, K. aerogenes, K. pneumoniae isolates (NCBI BioSample/SRA) | none | coverage of antibiotic resistance gene clusters (ARGC) scaled by gene length and sequencing depth | NCBI SRA / SRA Toolkit; DIAMOND alignment; USEARCH clustering |
| XGBoost machine learning classification | 915 public Gram-negative samples (five species, species-independent) | none | binary resistant vs susceptible label per antibiotic; MCC, accuracy, sensitivity, precision, specificity, AUC, PR-AUC | XGBoost (extreme gradient boosting) |
| Antimicrobial susceptibility testing (phenotype labels) | In-house CCHMC clinical isolates from blood/urine | none | resistant/susceptible/intermediate phenotype | VITEK 2 (Biomerieux) |
| Whole-genome sequencing (in-house) | 19 CCHMC clinical isolates (A. baumannii n=1, E. coli n=11, K. aerogenes n=2, K. pneumoniae n=3, K. oxytoca n=2) | none | paired 150 bp reads, ~5 million reads per sample | Illumina NextSeq 500; Nextera XT library kit; QIAamp PowerFecal DNA Kit |
- – 4933 of 6564 NCBI BioSample samples had required metadata; refined to 2516 for organisms/antibiotics of interest
- – Final balanced dataset of 946 samples; 915 used for model building/evaluation after setting aside 31 demo samples
- ▼ Clustering reduced input space from 4579 ARG to 1027 ARGC, with 410 clusters (40%) single-gene; only 152 ARGC found in data 4579 to 1027 (152 used)
- – Data split 70% training / 30% testing per antibiotic 70/30
- – For cefotaxime and ceftriaxone, under-sampling applied to reach 3:1 ratio to prevent overfitting 3:1
- – Sample counts per species: A. baumannii 256, E. coli 330, K. pneumoniae 211, E. cloacae 67, K. aerogenes 51
- count 6564 bacterial samples retrieved from NCBI BioSample (initial antibiogram-filtered samples)
- count 4933 samples with required metadata (after metadata filtering)
- count 2516 samples for organisms/antibiotics of interest (after refinement)
- count 946 final balanced samples; 915 used for modeling (after balancing; 31 set aside for demo)
- count 4579 antibiotic resistance genes; 1027 ARGC after clustering; 152 found in data (input feature space reduction)
- count 19 in-house CCHMC clinical samples (in-house validation dataset)
- count Ceftriaxone: 671 resistant, 133 susceptible, 804 total (per-antibiotic sample distribution (Table 1))
- count Ciprofloxacin: 577 resistant, 335 susceptible, 912 total (per-antibiotic sample distribution (Table 1))
Statistical methods review
Model: opusA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a machine learning study that trained and evaluated eight independent XGBoost (gradient-boosted decision tree) binary classifiers to predict resistant-versus-susceptible phenotype for eight antibiotics across five Gram-negative species, using antibiotic-resistance-gene coverage from WGS reads as input features. Models were trained on data randomly split 70% train / 30% test with 10-fold cross-validation and early stopping, and class imbalance was handled by balanced sampling and under-sampling to a 3:1 ratio. Performance was reported descriptively through classification metrics (Matthew's correlation coefficient, accuracy, sensitivity, precision, specificity, ROC-AUC, and precision-recall AUC) rather than through inferential hypothesis tests, and a reliability index based on a regression fit to misclassification rate was added.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| XGBoost (extreme gradient-boosted decision trees), binary classification | one independent model per antibiotic (8 models) predicting resistant vs susceptible | 915 samples total; per-antibiotic counts ranging 487 to 912 (Table 1) | stated |
| 10-fold cross-validation with early stopping | model training/hyperparameter evaluation for each of the 8 models | 70% training subset of available samples per antibiotic | na |
| Matthew's correlation coefficient (MCC) as performance statistic | evaluation of each model on imbalanced test data | 30% test subset per antibiotic | stated |
| Two-class accuracy, sensitivity, precision, specificity, ROC-AUC, precision-recall AUC | complete performance evaluation of each model | 30% test subset per antibiotic | na |
| Regression fit of misclassification rate vs adjusted model output (reliability index) | per-drug reliability index derived from test-subset predictions | test subset of each model | not stated |
-
Model performance was reported as point estimates of metrics (MCC, AUC, accuracy, etc.) on a single 30% held-out test split.↳ Could also: Reporting these metrics with confidence intervals via bootstrap resampling of the test set, or aggregating across repeated/nested cross-validation folds. — Interval estimates would convey the uncertainty around each performance value, which is especially informative when per-antibiotic test sets are modest in size.
-
Class imbalance was addressed through balanced sampling and under-sampling to a 3:1 ratio for the two most imbalanced drugs, with MCC used as the headline metric.↳ Could also: Alternative imbalance strategies such as class-weighting within XGBoost, oversampling (e.g., SMOTE), or threshold tuning could also be applied while retaining the full dataset. — Keeping all samples rather than discarding majority-class instances can preserve information, and comparing strategies would illustrate how robust the reported performance is to the resampling choice.
-
A single random 70/30 train/test split was used per antibiotic.↳ Could also: Repeated random splits or fully nested cross-validation could also be used to estimate generalization performance. — Averaging over multiple splits would reduce dependence of the reported numbers on one particular partition and give a sense of variance across partitions.
-
Coverage values from Eq. 1 were binarized to presence/absence (C>0 vs C=0) as model input.↳ Could also: The continuous coverage values could also be supplied directly as features. — Retaining the quantitative coverage could capture dose/abundance information that binarization removes, and comparing the two encodings would show whether that information aids prediction.
-
The reliability index was built by fitting a regression to the misclassification rate on each model's test subset.↳ Could also: Probability calibration approaches such as Platt scaling or isotonic regression, evaluated on a separate calibration set, could also be used to map model outputs to reliability. — A dedicated calibration set and established calibration methods would help ensure the reliability estimates generalize beyond the data used to fit them.
-
Models were trained pooled across species without stratification, intentionally to be species-independent.↳ Could also: Reporting per-species performance breakdowns alongside the pooled results could also be done. — A species-stratified summary would show how consistently the pooled model performs across the underrepresented species (e.g., K. aerogenes, E. cloacae) versus the well-represented ones.
Result convergence & founder nodes
Findings this paper shares with others that ran a comparable experiment. A node’s strength is how many independent papers report it (replication breadth) — not how often it is cited, so a heavily-replicated but under-cited founder still stands out.
-
ARG clustering compressed the feature space from 4579 antibiotic resistance genes to 1027 ARGCs; only 152 ARGCs were actually observed in the dataset, representing a ~97% reduction in effective features.WGS gram-negative bacteria down 2020×1papers★ This paper is the founder (earliest)
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
The inference pipeline reproduces exactly — all 8 antibiotic calls and reliability values for the shipped testFile.fastq.gz match the README's documented output 1:1, with byte-identical results from paired reads, indicating fully deterministic output and no fabrication concern for the shipped models. However, the comparison target is the repo's own example output, not the paper's reported scientific results: Table 3 accuracies (e.g. tobramycin 0.95) and Fig 3 AUC distributions were not reproducible because the training pipeline and 915-sample data were not shipped. The reproducibility friction is partly authors' side (undocumented mandatory version pins: diamond 0.8.36, xgboost 0.90; training code/data not deposited). Net: a solid but partial reproduction — exact on the inference, silent on the paper's central performance claims.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.