RiboTaxa: combined approaches for rRNA genes taxonomic resolution down to the species level from metagenomics data revealing novelties.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
PARTIAL, honest reproduction. The full RiboTaxa pipeline (third-party repo oschakoory/RiboTaxa v1.5 applied to the paper's own deposited MOCK data SRR5381736) was rebuilt FROM SCRATCH on «our HPC» this session — 3 conda envs (mamba), free usearch11, SILVA 138.1 NR99 -> matam @97% = 204,517 NR97 clusters (reproduced EXACTLY) -> sortmerna+bowtie indexes -> QIIME2 classifier — and run end-to-end twice (paper-default QC and a lighter-QC depth test). Reproduced vs Table 2: with DEFAULT QC genus P/R/F=93.8/78.9/85.7, species=92.9/65.0/76.5; with LIGHTER QC (read retention raised 56%->86%, near the paper's 92%) genus=94.4/89.5/91.9, species=80.0/60.0/68.6. PRECISION is near the reported 100% at both levels; the consistent shortfall is RECALL (genus ~79-90% vs reported 100; species 60-65% vs 85). Increasing depth recovered 2 of the 4 missed low-abundance genera, confirming depth matters for genus detection, but even at the paper's read depth the reported perfect genus and high species scores are NOT fully reproducible -> shortfall only partly explained by depth (residual: EMIRGE/MetaRib stochasticity, sortmerna 2.0 vs 2.1b, usearch build, or a more lenient paper detectability/metric definition). DB construction and all dataset Ns reproduce exactly. NOT attempted: Table 3 synthetic-gut (reads undeposited, simulator unspecified) and PRJEB13222 reprocessing (real data, no ground truth). All grades PROVISIONAL; raw taxonomy TSVs in reproduction/outputs/ for human re-derivation.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 68assessed: 2026-06-22 ⛓ 7a7b9d4f3bd3
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-29
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-22no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe authors aimed to identify which combination of rRNA read mappers, reference-targeted assemblers and taxonomic classifiers can achieve accurate taxonomic resolution down to the species level from shotgun metagenomics data, and to compile the best-performing tools into a single pipeline (RiboTaxa).
- ★ RiboTaxa, combining BBTools, FastQC, SortMeRNA, MetaRib, EMIRGE, VSEARCH, BBMap and QIIME 2's Sklearn classifier, was built as a pipeline for SSU rRNA-based taxonomic profiling of metagenomics data. method
- ★ RiboTaxa gave the best results compared to other tools (Kraken2, Centrifuge, METAXA2, phyloFlash, SPINGO, BLCA, MEGAN) with precise taxonomic identification and relative abundance description without false positive detection. finding
- ★ Applied to real datasets from ocean, soil and human gut environments and to gene-capture-by-hybridization datasets, RiboTaxa revealed microbial novelties not discerned by current bioinformatics analyses. finding
- ★ Unassembled short-length reads do not contain all informative regions of the 16S rRNA gene, reducing taxonomic assignment accuracy at species level, whereas converting reads into full-length or near-full-length rRNA sequences yields more detailed taxonomic resolution. finding
- METAXA2 is a hybrid pipeline that extracts rRNA reads from large sequencing datasets using Hidden Markov Models and then performs a BLAST search against a specialized rRNA database, reporting hierarchical classification down to genus or species level. mechanism
- EMIRGE uses a Bayesian approach to iteratively map short Illumina reads against reference gene sequences with Bowtie to reconstruct full-length rRNA sequences. mechanism
- MetaRib uses the same iterative mapping algorithm as EMIRGE but adds sub-assembly and dereplication before iterative mapping to reduce running time and memory usage. mechanism
- ★ The RiboTaxa workflow trims reads with bbduk.sh, reconstructs SSU rRNA sequences with MetaRib (unfiltered reads) and EMIRGE (SortMeRNA-filtered reads), estimates abundance by mapping reads back with BBMap, clusters sequences at 97% with VSEARCH, and classifies them with QIIME 2's sklearn plugin. method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| shotgun metagenomic sequencing + quality control/trimming | microbial MOCK community (21 bacterial, 7 archaeal species) | none | read quality metrics before/after trimming | BBDuk/BBTools v37.62, FastQC v0.11.9, Illumina MiSeq 2x300bp |
| SSU rRNA read extraction/mapping | microbial MOCK community shotgun reads | none | proportion and identity of aligned SSU rRNA reads | BBMap v38.87, SortMeRNA v2.1b against SILVA 138.1 database |
| full-length SSU rRNA gene assembly/reconstruction | microbial MOCK community reads | none | reconstructed nearly full-length SSU rRNA sequences and relative abundances | EMIRGE v0.61.1, MetaRib, MATAM v1.6.0 |
| taxonomic classification/affiliation | reconstructed SSU sequences from MOCK community | none | taxonomic assignment accuracy (precision/F-measure) at genus and species level | Kraken2 v2.0.8-beta, Centrifuge v1.0.3-beta, mothur v1.33, SPINGO v1.3, BLCA v2.1, QIIME 2 Sklearn classifier, RDP Classifier v11.5, blastn NCBI BLAST+ v2.11.0 |
| in silico synthetic metagenomic read simulation | synthetic complex human gut microbiota (100 microorganisms) | none | simulated 20X coverage paired-end Illumina reads and theoretical species abundance | ART simulator v2.5.8, Barrnap v0.9, SortMeRNA, VSEARCH |
| shotgun metagenomics taxonomic profiling with RiboTaxa pipeline | real environmental and host-associated datasets (ocean, soil, human gut) and gene-capture-by-hybridization datasets | none | taxonomic identification and relative abundance, detection of microbial novelties | RiboTaxa pipeline (BBTools, FastQC, SortMeRNA, MetaRib, EMIRGE, VSEARCH, BBMap, QIIME 2 Sklearn) |
- – RiboTaxa achieved an F-measure of 100% at the genus level compared to other benchmarked tools 100%
- – RiboTaxa achieved an F-measure of 83.7% at the species level compared to other benchmarked tools 83.7%
- – Application of RiboTaxa to real ocean, soil and human gut metagenomics datasets, as well as gene capture by hybridization datasets, revealed microbial novelties not detected by current bioinformatics pipelines
- – Short, unassembled reads limit taxonomic affiliation typically to family or genus level, while full-length reconstruction improves resolution to species/strain level
- other F-measure = 100% (RiboTaxa taxonomic accuracy at genus level vs other tools)
- other F-measure = 83.7% (RiboTaxa taxonomic accuracy at species level vs other tools)
- count 21 bacterial and 7 archaeal species (composition of the microbial MOCK community)
- count 100 microorganisms (synthetic complex human gut microbiota community)
- other 2 × 300 bp paired-end MiSeq sequencing (shotgun sequencing of the MOCK community (accession SRR5381736))
- other 20X coverage, 250 bp reads (ART simulator parameters used to generate synthetic human gut metagenomic reads)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a computational benchmarking study rather than a hypothesis-testing biological experiment: the authors evaluated read mappers, assemblers, and taxonomic classifiers on a microbial mock community and a synthetic complex human-gut community, then built a pipeline (RiboTaxa) from the best-performing tools. Performance was reported using classification/assembly performance metrics (e.g. F-measure, precision, taxonomic resolution, false positive detection) compared descriptively across tools, rather than through inferential statistical hypothesis tests.
-
Tool and pipeline performance was assessed using a single microbial mock community and a single simulated synthetic human-gut community, with results reported as point estimates (e.g. F-measure, precision).↳ Could also: Generating multiple replicate synthetic communities (varying random seeds, coverage, or community composition) and reporting variability across replicates (e.g. mean ± SD/range of F-measure) — Replicated simulations would allow the variability of performance metrics to be quantified and would let readers gauge how sensitive the tool rankings are to the specific dataset used.
-
Differences in performance between tools (e.g. RiboTaxa versus Kraken2, Centrifuge, METAXA2, phyloFlash, SPINGO, BLCA, MEGAN) were presented as descriptive comparisons of metrics like F-measure without formal inferential statistics.↳ Could also: Applying a statistical test appropriate for paired classifier comparisons on a shared set of reads/taxa, such as McNemar's test or a bootstrap-based confidence interval on the difference in F-measure — A formal test or interval estimate would quantify whether observed differences in classifier performance exceed what could arise from sampling variability alone, complementing the descriptive metric comparison already presented.
-
Relative abundance estimates from RiboTaxa and other classifiers were compared to theoretical/expected abundances derived from the mock and synthetic communities.↳ Could also: Reporting a correlation coefficient (e.g. Spearman or Pearson) or a Bland-Altman style agreement analysis between observed and expected relative abundances — A quantitative agreement statistic would summarize how closely estimated abundances track expected values across all taxa in a single interpretable number, complementing the qualitative/tabular abundance comparisons described.
-
Taxonomic classification confidence was fixed at a single cut-off (0.7) across all classifiers compared.↳ Could also: Reporting classifier performance (e.g. precision-recall or ROC-style curves) across a range of confidence thresholds — Threshold-sweep analyses would show how sensitive the comparative conclusions are to the chosen confidence cut-off, complementing the single-threshold comparison presented.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-36159175 (RiboTaxa)
Paper: Chakoory O, Comtet-Marre S, Peyret P. RiboTaxa: combined approaches for rRNA genes taxonomic resolution down to the species level from metagenomics data revealing novelties. NAR Genom Bioinform 4(3):lqac070 (2022). DOI 10.1093/nargab/lqac070 · PMCID PMC9492272.
Code (CORRECTED): The scaffold's code_url
(github.com/timoast/af73c0e9fac00187ee49) is wrong — it returns HTTP 404 and
is an unrelated auto-harvested gist hash (timoast = Tim Stuart, Signac author).
The real pipeline repo per the paper's Data & Code Availability statement is
https://github.com/oschakoory/RiboTaxa (RiboTaxa v1.5, bash + Python 2.7/3.6).
What RiboTaxa is
A bash pipeline that filters + reconstructs full-length SSU rRNA (16S/18S) genes from Illumina shotgun metagenomics and classifies them to species level. Steps & tools: FastQC/MultiQC (QC) → BBTools (trim/adapter) → SortMeRNA (rRNA filtering) → EMIRGE (emirge_amplicon.py) + MetaRib (reconstruction, then VSEARCH/BBMap cluster at 97% NR97) → QIIME2 sklearn classifier (confidence 0.7) against SILVA SSU 138.1 (released 2020-08-27).
Datasets the paper relies on
| Dataset | Accession | Type | N | Role | Ground truth? |
|---|---|---|---|---|---|
| MOCK microbial community | SRR5381736 | 2×300 MiSeq shotgun | 1 run, 28 species (21 bact + 7 arch); 1,246,376 read pairs (1,150,991 after filter) | Table 2 benchmark | YES (known 28-sp composition) |
| Synthetic human gut | none (simulated from 100 NCBI genomes) | simulated 250 bp PE, 20X | 100 species / 45 genera; 14,754,123 hq pairs | Table 3 benchmark | YES (known 100-sp composition) |
| Ocean (corals/sediment/seawater) | PRJEB13222 | shotgun | 20 samples, 21.07 Gb, 101 bp | real-data application (Table 1) | NO |
| Human gut age groups | PRJNA553191 | shotgun | 62 samples, 151.16 Gb | real-data application | NO |
| Soil / permafrost | PRJNA647119 | shotgun | 12 samples, 23.04 Gb | real-data application | NO |
| 16S gene-capture contaminated soil | SRR3648004 | targeted capture PE | 1 sample, 1.2 Gb | real-data application | NO |
In scope (pipeline-derived, reproducible)
- R0 — pipeline executes & reproduces shipped
test_dataexpected output (repo shipstest_data/with example config + expected*_SSU_taxonomy_abundance.tsv). QUICK FLOOR / sanity that the pipeline runs end-to-end on «our HPC». - R1 — MOCK community SRR5381736 → Table 2 RiboTaxa metrics (precision/recall/F-measure at genus & species: reported 100/100/100 genus, 100/85/91.9 species). Cleanest falsifiable target: real deposited data + known 28-species ground truth + reported numbers. PRIMARY 1:1 target.
- R2 (harder) — Synthetic human gut → Table 3 (100/100/100 genus; 100/72/83.7 species, the abstract headline). Requires re-simulating reads from the 100 genomes (the genome list must be recovered from supplementary); the simulation itself is a reproduction step → higher effort, attempt after R1.
- R3 (qualitative) — real datasets (PRJEB13222 etc.): run pipeline, compare taxonomy/abundance + "novelties". No ground truth → not a 1:1 metric; profile the data and attempt a descriptive run if compute allows.
Out of scope
- Wet-lab / manual novelty interpretation & phylogenetic-tree curation.
- Comparator tools' own numbers (MATAM, phyloFlash, METAXA2, Kraken2, MEGAN6, BLCA) except as context — we reproduce RiboTaxa's own outputs, not re-benchmark every competitor (would be reproduced only if cheap).
Known reproducibility risks (flag for audit)
- EMIRGE requires USEARCH (proprietary, licensed binary) per the repo README ("USEARCH must be installed in system PATH for EMIRGE functionality"). No USEARCH license ⇒ the EMIRGE branch may not run. MetaRib (BBMap-based, free) is the fallback reconstruction; need to check whether the pipeline degrades gracefully to MetaRib-only and how that affects the reported metrics.
- EMIRGE is Python 2.7,
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
Partial but honest reproduction. Input data identity is strong (mock SRR5381736 = 1,246,376 pairs exactly; ocean PRJEB13222 = 20 samples) and genus precision reproduced exactly (100%, 0 false positives) with the R0 shipped test data within tolerance — no fabrication signal. The shortfall is on our/technical side: a QC read-survival gap (56% vs the paper's 92%, same bbmap 39.01 + documented params) under-reconstructed low-abundance taxa, dropping recall (genus 78.9 vs 100; species 65 vs 85; species F 70.27 vs 91.9). The abstract headline (synthetic-gut species F=83.7%) is untested because the read simulator is unspecified (data_unavailable, not an authors' defect on the core mock claim). Net: magnitude/direction and the species-resolution concept hold, so overall yellow.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.