Graph-Based Approaches Significantly Improve the Recovery of Antibiotic Resistance Genes From Complex Metagenomic Datasets.
The main results reproduced: recomputed values matched the published ones within tolerance.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No authors-side cause for any deviation
- 🟡A deviation arose in the data or preprocessing
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
REPRODUCED (partial, strong within-tol). The paper is fully computational (GraphAMR: metaSPAdes -> PathRacer AMR-HMM-to-graph -> mmseqs2 ORF clustering @0.90/0.95/1.0 -> abricate ncbi). Reproduced 1:1 on the BRIEF accession SRR5997540 («our HPC» «job», metaSPAdes 3.15.0 + pathracer 3.16 + mmseqs2 + abricate 1.0.1 = the GraphAMR-pinned versions & params). Table 1 AMR counts agree within ~6%: contigs 93 vs 92, ORF@90/95/100 = 99/101/127 vs 103/105/135; graph HMM-paths 154 vs 169; Table 3 graph-based count 88-91 vs 90. The HEADLINE qualitative claim is CONFIRMED: graph (154) > contig (93), and graph/ORF counts (88-127) >> read-based SRST2 (reported 59). Residual deltas are explained by AMR-database version drift (paper: CARD v3.0.8/v3.1.2 + 2021 NCBI AMR HMMs; ours: current NCBI AMRFinder AMR.LIB 580 HMMs + abricate-1.0.1 ncbi 2025-Jan-14) plus metaSPAdes graph nondeterminism — NOT a discrepancy in method. NOT attempted: SRST2 read baseline (T3-srst2) and CARD-AA db (T2-AA) — secondary comparison numbers; case-study figures (blaIMP/blaCTX-M graph layouts) are illustrative, not graded; the other 12 URBAN runs + SOIL SRR8931193 are extensions. Getting here required diagnosing 3 failures: pathracer-segfault on a 4.x GFA (fixed by pinning spades 3.15.0), the SPAdes-3.15 pyyaml3/python-3.12 collections.Hashable crash (fixed by pinning python=3.9), and a transient «infra» user-quota exhaustion (worked around by running heavy I/O on node-local scratch). No fabrication concern: abricate output contains real, plausible ARGs and reproduced counts track the paper.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-18 ⛓ c9d25804631b
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-25
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetGraph-based alignment of AMR profile HMMs directly to metagenomic assembly graphs (rather than to assembled contigs/scaffolds or raw reads) will significantly improve the sensitivity, accuracy, and completeness of antibiotic resistance gene recovery from fragmented, complex metagenomic datasets.
- ★ GraphAMR, a Nextflow pipeline that aligns AMR profile HMMs (or AA sequences) to metagenomic assembly graphs via PathRacer, then dereplicates and annotates hits, recovers more and more complete AMR genes than contig-based or read-based methods. resource
- ★ AMR gene sequences are frequently fragmented across multiple edges/contigs/scaffolds of a metagenomic assembly graph due to interspecies repeats, strain variation, and horizontal gene transfer, causing contig-based tools to miss or misidentify them. mechanism
- ★ The blaIMP beta-lactamase gene, though absent from Ng et al. (2017)'s read-based resistome and absent from assembled scaffolds, is present in the sample but scattered over 10 edges and 2 scaffolds of the assembly graph. finding
- ★ The blaCTX-M-9 and blaCTX-M-14 gene variants (differing by only 2 amino acids) cannot be resolved by the assembler into complete contigs, unlike the blaCTX-M-15 variant which resides on a single contig. finding
- ★ More than 30% of all AMR profile HMM hits to the assembly graphs of Ng et al. (2017) samples are not contained within a single scaffold. finding
- ★ Graph topology (isolated short edges) and hit-length coverage can be used as filters to distinguish trustworthy AMR hits from spurious ones. method
- 93% of genes in the NCBI AMR database are longer than 300 bp, exceeding typical short-read lengths (100-300 bp), limiting read-based AMR detection methods. finding
- GraphAMR supports both profile HMM alignment (default NCBI AMR database) and amino-acid sequence alignment (e.g., CARD, ResFinder) to the assembly graph via a proxy-HMM/BLOSUM62 approach. method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Profile HMM alignment to assembly graph (PathRacer) | URBAN wastewater/urban surface metagenomes (Ng et al. 2017), samples SRR5997540-SRR5997552 | none | fraction of HMM hits not residing on a single scaffold | PathRacer |
| Metagenomic de novo assembly + AMR gene prediction (Abricate) | URBAN wastewater metagenomes, 13 samples | none | predicted AMR gene counts from contigs vs. unclustered HMM paths vs. clustered ORFs (90%, 95%, 100% IDY) | metaSPAdes, Abricate |
| AMR gene prediction using amino-acid sequence alignment to assembly graph | URBAN metagenomes | database choice (CARD v3.1.2 AA vs. NCBI AMR HMM) | unique AMR gene sequence counts | PathRacer, Abricate |
| Read-based AMR gene detection (SRST2) vs. graph-based (GraphAMR) | URBAN metagenomes, raw reads | none | predicted unique AMR gene counts, using shared CARD_v3.0.8_SRST2 database | SRST2, GraphAMR/Abricate |
| AMR gene prediction from assembly graph vs. assembled scaffolds | SOIL groundwater metagenome sample SRR8931193 (Smith et al. 2019) | none | AMR gene counts and identity (e.g., vanR-O, ant(6)-Ib, tet(X)) | GraphAMR/Abricate |
| Bioinformatic database survey | NCBI AMR database (Feldgarden et al. 2019) gene entries | none | distribution of AMR gene lengths | — |
- – In sample SRR5997545, contig-based prediction yielded more AMR genes (9) than HMM-path-based prediction (8), traced to a short hit covering only 73% of HMM length on an isolated edge below the default 90% coverage threshold.
- ▲ Across nearly all URBAN samples, GraphAMR predicted more unique AMR genes than SRST2 (e.g., sample 40: 90 vs. 59; sample 48: 82 vs. 59) when using the same CARD_v3.0.8_SRST2 database.
- ▼ In sample SRR5997542 (id 42), SRST2 predicted 5 more AMR genes than GraphAMR, but these were found to be likely spurious, arising from fragmented sequences on isolated graph edges.
- – In SOIL sample, Abricate predicted 12 AMR genes from clustered HMM paths vs. 13 from assembled scaffolds; vanR-O and ant(6)-Ib were found only on scaffolds, while tet(X) was detected only by GraphAMR.
- – The ant(6)-Ib gene sequence in the SOIL sample is split across two isolated edges in the assembly graph, and the vanR-O hit covered only 30% of the corresponding sequence, likely spurious. 30% coverage
- ▼ Unclustered HMM paths consistently show higher counts than dereplicated/clustered ORFs (e.g., sample 40: 169 HMM paths vs. 103 clustered ORFs at 90% IDY), reflecting redundant paths from synonymous mutations or node-terminated alignments.
- – More than 30% of all NCBI-AMR HMM hits aligned to URBAN sample assembly graphs are not contained on a single scaffold. >30%
- other 93% (proportion of AMR genes in NCBI AMR database longer than 300 bp)
- other >30% (fraction of total HMM alignment hits to assembly graphs not residing on a single scaffold, URBAN dataset)
- count 741 bp / 246 amino acids (length of blaIMP gene cassette)
- count blaIMP scattered over 10 edges and 2 scaffolds (assembly graph representation of blaIMP gene)
- count GraphAMR 90 vs. SRST2 59 (sample 40) (AMR gene counts comparison, URBAN sample SRR5997540)
- count SOIL: 12 (clustered HMM paths) vs. 13 (assembled scaffolds) (AMR gene counts, SOIL groundwater sample SRR8931193)
- other vanR-O hit covered only 30% of sequence (coverage of spurious hit, SOIL sample)
- count SRR5997545: contigs 9 vs. HMM paths 8 (outlier sample, coverage threshold effect (73% HMM coverage below 90% default))
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a computational methods paper presenting GraphAMR, a graph-based pipeline for antibiotic resistance gene (ARG) recovery from metagenomic assembly graphs. Performance was evaluated descriptively by comparing raw counts of predicted ARG sequences across approaches (contig-based, unclustered HMM paths, clustered ORFs at 90/95/100% identity, and read-based SRST2) on publicly available environmental metagenome datasets. No inferential statistical tests were applied; conclusions about improvement were drawn from tabular count comparisons and visual inspection. All results were reported as absolute predicted gene counts per sample with no summary statistics or uncertainty measures.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| descriptive count comparison (no formal statistical test) | Comparison of ARG counts from assembled contigs, unclustered HMM paths, and clustered ORFs at 90/95/100% AA identity (Tables 1–2) | 13 urban wastewater metagenome samples (SRR5997540–SRR5997552) | na |
| descriptive count comparison (no formal statistical test) | GraphAMR (graph-based, clustered HMM paths) vs. SRST2 (read-based) predicted ARG counts (Table 3) | 13 urban wastewater metagenome samples | na |
| descriptive proportion | Fraction of HMM hits not residing on a single scaffold, used to motivate the graph-based approach (Figure 4) | 13 urban wastewater metagenome samples | na |
-
Performance was assessed by comparing absolute ARG counts per sample across tools, with conclusions drawn from visual inspection of tables↳ Could also: Precision, recall, and F1 score against a curated ground truth (e.g., from simulated reads with known ARG complement) could also quantify performance — A recall/precision framework would distinguish true positives from spurious predictions and allow single-number performance summaries comparable across future studies, rather than requiring sample-by-sample count inspection
-
The method comparison was conducted exclusively on real environmental metagenomes with no known ground truth for ARG content↳ Could also: Synthetic benchmark datasets (reads simulated in silico from genomes carrying known ARGs) could also serve as a complementary evaluation set — Simulated data with a defined ground truth enables direct computation of sensitivity and specificity, providing a controlled complement to real-data experiments
-
Count differences between GraphAMR and SRST2 across 13 paired samples were described sample-by-sample without a formal test of consistency↳ Could also: A paired non-parametric test (e.g., Wilcoxon signed-rank) across the 13 samples could also summarize whether the count advantage is systematic — A paired test would express whether higher GraphAMR counts are a consistent pattern across samples or are driven by a small number of high-count samples, enabling an inferential rather than purely descriptive statement
-
ARG counts were compared as raw numbers without normalizing for sequencing depth or total read count↳ Could also: Depth-normalized metrics (e.g., ARGs per million reads or per Gbp of sequence) could also be reported alongside raw counts — Depth normalization makes cross-sample comparisons less sensitive to differences in library size and aligns with common practice in comparative metagenomics
-
Three clustering thresholds (90%, 95%, 100% AA identity) were applied and resulting counts tabulated, but no formal criterion was stated for selecting 90% as the primary threshold↳ Could also: A systematic threshold sweep (e.g., 70–100% in 5% steps) with a stated selection criterion could also characterize how count estimates depend on dereplication stringency — A sensitivity analysis would help readers understand result stability across a range of clustering parameters and guide threshold selection in future applications
-
Across-sample performance of GraphAMR was summarized by listing per-sample counts without aggregating over the 13-sample URBAN collection↳ Could also: Summary statistics (median and IQR, or mean and SD of per-sample counts or fold-differences) could also characterize the central tendency and variability of the improvement across samples — Aggregate summaries would give readers a single interpretable estimate of the typical gain from the graph-based approach rather than requiring mental averaging over 13 columns
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-34690959
Title: Graph-Based Approaches Significantly Improve the Recovery of Antibiotic
Resistance Genes From Complex Metagenomic Datasets (Shafranskaya, Chori,
Korobeynikov, Front Microbiol 2021).
Code: https://github.com/ablab/graphamr (GraphAMR — Nextflow DSL2 pipeline)
Primary data accession (this RU): SRR5997540 (URBAN wastewater metagenome,
sample "40"; one of 13 runs SRR5997540–SRR5997552 from Ng et al. 2017).
The whole paper is computational (no wet-lab component)
GraphAMR is a bioinformatics pipeline. Every reported number is pipeline-derived, so in principle everything is in scope. We focus reproduction effort on the accession named in the BRIEF, SRR5997540, because (a) it is the canonical example sample carried through Tables 1–3 and the case studies, and (b) the full pipeline is heavy (metaSPAdes assembly + PathRacer graph alignment), so one sample end-to-end is the realistic minimum; the other 12 URBAN samples and the SOIL sample (SRR8931193) are extensions if compute budget allows.
Pipeline (per README + Methods)
reads → FastQC QC → metaSPAdes assembly (produces GFA assembly graph) →
PathRacer (aligns AMR profile-HMMs to the assembly graph, emits up to top-100
non-redundant paths/edges) → MMseqs2 dereplication (complete ORFs clustered at
90 / 95 / 100 % AA identity) → annotation (Abricate, RGI, sraX) →
hAMRonization summary. Default HMM DB = NCBI AMR profile HMMs; alternative =
CARD v3.0.8 / v3.1.2 AA. Read-based baseline for comparison = SRST2.
Orchestrated by Nextflow (nextflow run ablab/graphamr -profile <conda> ...).
IN SCOPE (pipeline-derived results to reproduce for SRR5997540)
| id | result | reported (sample "40" = SRR5997540) | paper loc |
|---|---|---|---|
| T1-contigs | Abricate AMR hits on assembled contigs/scaffolds | 92 | Table 1 |
| T1-hmmpaths | Abricate AMR hits on PathRacer HMM paths (graph) | 169 | Table 1 |
| T1-orf90 | Abricate AMR hits, ORFs clustered @90% AA | 103 | Table 1 |
| T1-orf95 | … @95% | 105 | Table 1 |
| T1-orf100 | … @100% | 135 | Table 1 |
| T2-AA | hits vs CARD AA db | 96 | Table 2 |
| T2-HMM | hits vs NCBI AMR HMM db | 94 | Table 2 |
| T3-srst2 | read-based ARG count (SRST2) | 59 | Table 3 |
| T3-graphamr | graph-based ARG count (GraphAMR) | 90 | Table 3 |
| F4-pct | share of HMM hits NOT contained in a single scaffold | ">30%" | Fig 4 / text |
Primary success criterion: regenerate the GraphAMR graph-based count (Table 3 = 90,
and/or Table 1 HMM-paths = 169 / ORF@90 = 103) for SRR5997540 and show the
qualitative finding (graph > contig and graph > read-based) holds. Exact-integer
agreement is unlikely given tool-version drift (no pinned commit, CARD version
matters, metaSPAdes nondeterminism), so grading expects within-tol/partial
with the direction of improvement as the robust claim.
OUT OF SCOPE / not attempted
- Wet-lab: none (paper is fully computational).
- Manual case-study figures (blaIMP edge layout, blaCTX-M contig scatter) — these are illustrative graph visualisations, not single reproducible numbers; noted but not graded.
- SOIL sample SRR8931193 and URBAN samples other than SRR5997540 — extensions, attempted only if «our HPC» compute budget permits after the primary sample.
Known reproduction risks
- Repo has no release/pinned commit →
pinned_refresolved at clone time (P-risk: env_unresolvable if branch moved). Will record the exact SHA used. - ARG counts depend on the CARD/NCBI-AMR DB version; paper used CARD v3.0.8 / v3.1.2 and "NCBI AMR HMMs" of 2021. Current DB versions will differ → count drift.
- metaSPAdes assembly is sensitive to thread count / version → graph differs → downstream counts differ. Not bit-reproducible.
- Heavy compute (metaSPAdes on a wastewater metagenome) → «our HPC» SLURM only.
Current blocker (2026-06-19)
«our HPC» SSH tunnel is DOWN ("Connection timed out during banner exchange")
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.