Tourmaline: A containerized workflow for rapid and iterable amplicon sequence analysis using QIIME 2 and Snakemake.
Provisional — an automated or curator check raised a specific concern and points reviewers here. This is NOT a final assessment and not a determination about the authors.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- 🟡A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
PARTIAL (1:1 on the workflow + dataset; numeric headline differs and is diagnosed). Tourmaline = authors' own QIIME2 2021.2 / Snakemake amplicon workflow, pinned to paper-era commit 8dbf859. ENV + DATA verified: all 96 16S V4 runs of PRJNA679730 downloaded, total reads 11,552,492 vs paper 11,552,448 (match, C2a). T1 FLOOR exact: shipped 16-sample test = 301 ASVs, bit-identical across two runs (deterministic). HEADLINE: running the SHIPPED config (DADA2-PE trunc 240/190, trim_left=0) on the deposited reads gives 17,646 ASVs / length avg 315 vs paper 12,379 / 258 -> C1 mismatch, C2 partial (min/max bounds 240/416 reproduce; avg shifted longer). DIAGNOSED: the SRA-deposited reads still carry 5' heterogeneity spacers + EMP 515F/806R primers (primer found ~13bp into reads, not at start), which the shipped config's trim_left=0 does NOT remove -> inflates ASV count + merged length. The paper's avg 258 (~clean V4 amplicon) implies an undocumented primer-trim step. Follow-up «job» runs cutadapt primer removal then identical DADA2 to test whether it recovers ~12,379/258. NOT attempted 1:1: Table 1 runtimes (hardware-specific); C3-C5 taxonomy/diversity (stretch).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 61assessed: 2026-06-21 ⛓ 207821248bad
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-21
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-22no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetExisting amplicon sequencing analysis tools (like QIIME 2) lack automation and standardization across runs, creating inefficiency and barriers to meta-analysis; a Snakemake-based workflow wrapping QIIME 2 with a standardized directory structure, automated reporting, and iterability can make amplicon data analysis faster, more reproducible, and easier to share.
- ★ Tourmaline is a Python-based workflow that implements QIIME 2 using the Snakemake workflow management system to automate amplicon sequence analysis. method
- ★ Automated workflows like Tourmaline enable rapid analysis of environmental amplicon data, decreasing time from data generation to actionable results. finding
- ★ Tourmaline's defined, reusable directory structure facilitates comparison of runs with different parameters and supports meta-analysis across studies. mechanism
- ★ Tourmaline generates an HTML report summarizing metadata, outputs, and linking to interactive QIIME 2 visualizations and Jupyter notebooks. method
- Tourmaline supports both DADA2 (paired-end/single-end) and Deblur (single-end) denoising algorithms plus multiple taxonomic classification and alignment methods. method
- Tourmaline runs natively on Linux/macOS or via a Docker container, avoiding dependency issues across platforms. resource
- ★ Application to a downsampled 16S rRNA gene data set from Western Lake Erie surface water demonstrates Tourmaline's utility for parameter optimization and interactive visualization. finding
- Tourmaline is freely available at github.com/aomlomics/tourmaline. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| 16S rRNA gene amplicon sequencing (metabarcoding) | Bacterial/archaeal community, Lake Erie surface water (environmental sample) | none | Taxonomic composition, feature/representative sequence properties, alpha and beta diversity | QIIME 2 (v2021.2) via Snakemake, DADA2/Deblur denoising |
- – Test data set (subsampled to 1,000 sequences/sample) allowed the entire Tourmaline workflow to run in approximately 10 minutes.
- – Workflow defines 21 pseudo-rules (3 denoising methods x 7 pseudo-rules each) and 47 regular rules in the Snakefile. 21 pseudo-rules; 47 regular rules
- – Tourmaline produced representative sequence property tables/plots, taxonomy barplots, alpha/beta diversity outputs, and an HTML report for the Lake Erie test data set, demonstrating parameter optimization utility.
- count 1,000 sequences per sample (Subsampling depth of the Lake Erie test data set)
- other ~10 minutes (Runtime for entire workflow on test data set)
- count 21 pseudo-rules (3 denoising methods x 7 pseudo-rules each defined in Snakefile)
- count 47 regular rules (Rules defined in Snakefile performing QIIME 2, Python, and shell commands)
- count 4 alpha diversity metrics (Observed features, Faith's PD, Shannon diversity, Pielou's evenness)
- count 4 beta diversity metrics (Unweighted/weighted UniFrac, Jaccard distance, Bray-Curtis distance)
- other QIIME 2 version 2021.2 (Required QIIME 2 version for Tourmaline installation)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a technical note describing a bioinformatics software workflow (Tourmaline) rather than a hypothesis-driven experimental study. The paper demonstrates the pipeline on a test 16S rRNA amplicon dataset from Lake Erie surface water, running QIIME 2-based denoising, taxonomic assignment, and alpha/beta diversity analyses that include built-in 'group significance' tests, with results presented through QIIME 2 visualization files (.qzv) and an HTML report rather than as a set of hypothesis-test results reported in the text itself.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Alpha diversity group significance test (specific statistic, e.g. Kruskal-Wallis, not named in the provided text) | Comparison of alpha diversity metrics (observed features, Faith's PD, Shannon diversity, Pielou's evenness) across metadata groups, output as observed_features_group_significance.qzv and equivalents | — | not stated |
| Beta diversity group significance test (specific statistic, e.g. PERMANOVA/ANOSIM, not named in the provided text) | Comparison of beta diversity distances (unweighted/weighted UniFrac, Jaccard, Bray-Curtis) across a single metadata column, output as beta_group_significance.qzv | — | not stated |
| Outlier detection (odseq) | Representative sequence curation during the diversity step | — | not stated |
-
Alpha diversity group significance is run via QIIME 2's built-in test without the specific statistic (e.g., Kruskal-Wallis) being named in the text.↳ Could also: Explicitly reporting the test name along with checked assumptions (e.g., distributional shape of diversity indices) could also be done — This would let readers judge whether a nonparametric or parametric comparison is more appropriate for a given diversity metric and sample size.
-
Beta diversity group significance is assessed for a single metadata column per run using distance-based methods (UniFrac, Jaccard, Bray-Curtis).↳ Could also: A multivariate approach such as PERMANOVA with multiple covariates, or distance-based redundancy analysis (db-RDA), could also be used — These approaches can partition variance across several metadata variables simultaneously rather than testing one column at a time.
-
Four alpha diversity metrics and four beta diversity metrics are each tested for group significance in a given run.↳ Could also: Applying a multiple-testing correction (e.g., Benjamini-Hochberg FDR) across the panel of metrics tested per metadata column could also be reported — Since several related diversity metrics are tested together, a correction would help control the family-wise error rate across that set of comparisons.
-
Outlier detection in the workflow is performed using odseq during multiple sequence alignment curation.↳ Could also: Alternative outlier-detection approaches, such as robust compositional methods (e.g., via DEICODE/Aitchison distance) or median-absolute-deviation-based screening, could also be applied — Different outlier detection methods carry different assumptions about the data (e.g., alignment-based vs. compositional count-based), and comparing methods can show whether flagged sequences are sensitive to the chosen approach.
-
Beta diversity ordination includes UniFrac, Jaccard, Bray-Curtis distances alongside robust Aitchison PCA (DEICODE).↳ Could also: Additional log-ratio or compositional data analysis (CoDA) techniques could also be used for the count-based feature table — Amplicon feature tables are compositional (relative abundances), and log-ratio-based methods are a standard alternative for addressing compositionality directly in ordination and group comparisons.
-
The demonstration dataset was subsampled to 1,000 sequences per sample for speed, with sampling depth for diversity analyses chosen via alpha rarefaction curves.↳ Could also: Reporting rarefaction curve inflection points alongside a sensitivity analysis across a range of depths could also be included — This would make explicit how robust the group significance conclusions are to the chosen rarefaction/sampling depth, which is already a built-in workflow output but not detailed numerically in the text.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — PMID 35902092 (Tourmaline)
Paper: Thompson et al. 2022, Tourmaline: A containerized workflow for rapid and iterable amplicon sequence analysis using QIIME 2 and Snakemake. GigaScience 11:giac066. Code: https://github.com/aomlomics/tourmaline (authors' own tool — Snakemake wrapper around QIIME 2). Data: SRA BioProject PRJNA679730 (2018 Lake Erie 16S rRNA survey + other deposits).
What kind of paper this is
A software/workflow paper. Tourmaline is a Snakemake pipeline that orchestrates QIIME 2 (DADA2/Deblur denoise → representative sequences → taxonomy → diversity → HTML report). The paper demonstrates the tool on a real 16S V4 dataset and reports a handful of concrete pipeline outputs + a runtime benchmark table.
Version pinning (IMPORTANT)
- Paper describes QIIME 2 2021.2 and Tourmaline v1 (paired-end DADA2).
- All git release tags postdate the paper: v1.0.0 (2023-07, QIIME2 2023.2) … v2.0.0-beta (2025-11, QIIME2 2024.10).
- Repo HEAD today = Tourmaline 2 (different Snakefile structure, QIIME2 2024.10) — NOT the paper.
- Paper-era commit =
8dbf85968e17(2022-03-18), the last commit before publication (2022-07-29); its README + setup pin QIIME2 2021.2 and SILVA 138-99 515-806. This is the commit we reproduce against.
In scope (pipeline-derived, attempted)
| id | result | reported | pipeline |
|---|---|---|---|
| C1 | # representative sequences (ASVs), full 96-sample run | 12,379 | DADA2 PE (QIIME2 2021.2) |
| C2 | rep-seq length distribution | min 240, max 418, avg 258 | DADA2 PE → repseqs |
| C3 | dominant families per size fraction | 5.0 µm: Sporichthyaceae, SAR11 Clade III; 0.22 µm: Cyanobiaceae | taxonomy (consensus-vsearch, SILVA 138-99) |
| C4 | alpha diversity contrast | 5.0 µm fraction has greater within-sample diversity | diversity (alpha) |
| C5 | rarefaction sufficiency | ~350 sequences/sample sufficient | diversity (rarefaction/alpha-rarefaction) |
| T1 | shipped 16-sample test run is deterministic + completes | (no paper number; CI/example) | DADA2 PE on shipped 1000-seq/sample subsample |
C1 + C2 are the headline quantitative claims. T1 is the QUICK FLOOR (fully shipped data + code + params → deterministic, no download needed).
Out of scope (not attempted / not 1:1 comparable)
- Table 1 runtime benchmark (dada2_pe_denoise 02:05:43 @1 core / 00:38:10 @8 cores, etc.). Hardware-specific (18-core iMac Pro, 32 GB RAM). We can report our wall-clock on «our HPC» but it is NOT a faithful 1:1 comparison — different CPU/IO. Recorded as informational only.
- Wet-lab sampling / AUV-ESP collection, qPCR, DNA extraction — manual/wet-lab.
- The HTML report's rendering itself (software feature, not a numeric result).
Reproduction tiers
- Floor (T1): run shipped test data through DADA2-PE denoise + taxonomy + diversity; confirm the workflow runs end-to-end and is deterministic (run twice → identical feature IDs).
- Headline (C1/C2): identify the 96 Lake Erie 16S V4 samples in PRJNA679730, download on front1, run DADA2-PE with paper config (trunc-len-f 240, trunc-len-r 190, max-ee 2/2, trunc-q 2, independent pooling, consensus chimera) → compare ASV count to 12,379 and length dist to min 240 / max 418 / avg 258.
- Stretch (C3/C4/C5): taxonomy + diversity on the full run (qualitative comparison).
DADA2 PE params (config.yaml @ 8dbf859)
trunc_len_f=240, trunc_len_r=190, trim_left_f=0, trim_left_r=0, max_ee_f=2, max_ee_r=2,
trunc_q=2, pooling=independent, chimera=consensus, min_fold_parent_over_abundance=1,
n_reads_learn=1000000, hashed_feature_ids=on. classify_method=consensus-vsearch.
Reference: SILVA 138-99 515-806 (silva-138-99-seqs-515-806.qza, silva-138-99-tax-515-806.qza).
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
The input data is a 1:1 match (11,552,492 vs 11,552,448 reads) and the workflow is confirmed end-to-end runnable and deterministic (T1 bit-identical across two runs). The headline benchmark deviates — 17,646 ASVs / avg 315bp vs the paper's 12,379 / 258 — but this is cleanly diagnosed: the SRA reads still carry 5' spacers + EMP primers that the shipped trim_left=0 config never removes, implying an undocumented primer-trim step on the authors' side. This is a documentation/methodology gap, not fabrication: bounds (240/416) reproduce and the value is plausibly recoverable with cutadapt. Overall yellow — solid reproduction with an explainable, authors'-side deviation.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.