MEDUSA: A Pipeline for Sensitive Taxonomic Classification and Flexible Functional Annotation of Metagenomic Shotgun Sequences.
The main results reproduced, with only marginal, non-material deviations.
- ✓Reported values were directly comparable
- 🟡Could not use the authors’ exact input data
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
IN PROGRESS (not final). MEDUSA is a Snakemake metagenomics tool paper; its quantitative evidence is a MEDUSA-vs-MEGAN comparison reproducible from the dalmolingroup/ComparisonMEGANxMEDUSA repo (4 R scripts + Git-LFS classification tables, D1MEDUSA.txt=57MB). Plan: clone+git-lfs-pull on «infra», run the 4 authors' R scripts on «our HPC», compare regenerated MCC/sensitivity/GO-term counts to reported (species 95/51%, genus 99/78%, functional MCC 0.59/-0.22, GO terms 303/48). Full-pipeline re-run from FASTQ (DIAMOND vs NCBI nr, Kaiju) is the harder «our HPC»+big-DB target, attempted opportunistically. Execution-time benchmarks are out of scope (hardware-specific, not 1:1 reproducible). Blocker at start: «our HPC» SSH tunnel unreachable (central VPN); prep done offline, waiting on tunnel. NOTE: all grades provisional, must_be_independently_checked.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-19 ⛓ 9a3a84fda06f
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-19
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-07-31
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusBecause the choice of bioinformatics tools strongly affects the accuracy, formatting, and functional identifiers obtained in metagenomic shotgun analyses, can a benchmarked, automated pipeline be designed to deliver sensitive taxonomic classification and flexible functional annotation transferable to any user-defined functional identifier?
- ★ MEDUSA is an automated, Conda-installable and Snakemake-managed pipeline performing preprocessing, assembly, alignment, taxonomic classification, and functional annotation on shotgun data. resource
- ★ A novel Python tool (annotate) transfers annotations to BLAST/DIAMOND alignment results using user-built plain-text dictionaries, allowing reuse of one alignment to obtain any functional identifier. method
- ★ Compared with MEGAN 6 Community Edition, MEDUSA correctly identifies more species, especially less abundant ones, and is better suited for functional analysis using Gene Ontology identifiers. finding
- ★ Tool selection was driven by benchmarks: Fastp (preprocessing), Bowtie2 (decontamination), Kaiju (taxonomic classification, highest MCC), DIAMOND (alignment), and MEGAHIT (assembly). method
- Kaiju achieved the highest Matthews Correlation Coefficient at species and genus level among the taxonomic classifiers tested. finding
- In decontamination, BBMap and Bowtie2 were more sensitive to contaminants in scenarios with fewer human reads, while BWA-MEM produced more false positives and HISAT2 more false negatives. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Trimming/quality-control benchmark (read preprocessing) | Bovine fecal metagenome reads (SRA SRR5371509), 1/5/10/40 million reads, SE and PE | none | elapsed run time (averaged over 3 runs) using 1 and 4 cores | AfterQC, BBDuk, Cutadapt, Fastp, SOAPnuke, Trimmomatic, FastQC; phred threshold 20 |
| Decontamination/host-removal benchmark | Simulated datasets: human chr4 (RefSeq NC_000004.12) + 200 InSilicoSeq bacterial genomes, 4M reads at 25/50/75% human | none (in silico read simulation) | elapsed time, Matthews Correlation Coefficient, false negative and false positive counts | BBMap, Bowtie2, BWA, HISAT2; Ensembl Homo sapiens GRCh38 primary assembly v102 |
| Assembly benchmark | CAMI first-challenge low/medium/high complexity simulated datasets (CAMISIM) | none | assembly quality vs gold-standard assembly | MEGAHIT, MetaSPAdes; MetaQUAST |
| Taxonomic classification benchmark | Dataset 1: 509,688 Illumina MiSeq simulated reads (394 bacterial, 73 archaeal, 40 viral) with 99,918 negative-control 'unknown' reads | none (esl-shuffle of 199 bacterial sequences to simulate unknowns) | taxonomic assignment quality (MCC) | InSilicoSeq, HMMER v3.3; BASTA, Krona, Kaiju, Kraken 2, DIAMOND vs NCBI-nr |
| Functional annotation evaluation | Dataset 2: 400,433 InSilicoSeq reads from 10 bacterial organisms plus negative-control reads | none | transfer/accuracy of GO functional identifiers vs curated UniProtKB/Swiss-Prot annotations | InSilicoSeq, UniProt ID mapping API, R v4.0.5; annotate tool, DIAMOND vs NCBI-nr |
| Pipeline comparison (taxonomic + functional) on real data | Dataset 3: human gut metagenome from a Crohn's disease patient (SRR579292, BioProject PRJNA175224) | none | taxonomic and functional results vs MEGAN 6 CE | MEDUSA vs MEGAN v6.18.3 CE; phred 20, identity threshold 80%, NCBI-nr |
- – All decontamination tools achieved high MCC above 0.99, with BBMap and Bowtie2 most sensitive when human reads were fewer. MCC > 0.99
- – FastQC was fastest with one core; Fastp was the second-fastest single-core tool and fast enough with four cores while uniquely producing visual reports alongside FastQC.
- – BWA-MEM produced more false positives and HISAT2 produced more false negatives in host-sequence removal.
- ▲ Kaiju achieved the highest Matthews Correlation Coefficient at species and genus levels, motivating its selection for taxonomic classification.
- ▲ MEDUSA correctly identifies more species (especially less abundant) than MEGAN 6 CE and is better suited for GO-based functional analysis.
- other MCC > 0.99 (All decontamination tools' Matthews Correlation Coefficient in host-removal benchmark)
- count 509,688 reads (99,918 negative-control) (Dataset 1 for taxonomic classification benchmark)
- count 400,433 reads (Dataset 2 from 10 bacterial organisms for functional annotation)
- count 394 bacterial, 73 archaeal, 40 viral sequences (Source sequences for Dataset 1)
- other phred score threshold of 20 (Trimming threshold applied to all datasets)
- other 80% identity threshold (Percent identity threshold set for MEGAN and annotate comparison)
- count 200 bacterial genomes (InSilicoSeq genomes for decontamination dataset (seed 5))
- count 4 million reads (Reads per simulated decontamination dataset)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a bioinformatics methods paper introducing the MEDUSA pipeline; its 'statistics' are almost entirely performance metrics rather than inferential tests. Benchmarks compared tools on simulated and public datasets using elapsed time (averaged over three runs with the Unix time command) and the Matthews Correlation Coefficient (MCC) for classification quality. Comparisons between tools were made descriptively by inspecting figures and metric values; no formal hypothesis tests or p-values were reported.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Matthews Correlation Coefficient (MCC) — used as a scalar classification-quality metric, not a hypothesis test | Decontamination tools benchmark (Figure 3B/D) and taxonomic classification tools benchmark | Simulated datasets: decontamination dataset 4 million reads per scenario; taxonomic dataset D1 509,688 reads | na |
| Elapsed wall-clock time averaged over three runs (Unix 'time' command) | Trimming tools benchmark (Figure 2) and decontamination tools benchmark (Figure 3A/C) | 3 repeated runs per tool/dataset combination | not stated |
| False-negative and false-positive counts (descriptive enumeration) | Decontamination misclassification benchmark (Figure 4) | Simulated datasets with known ground truth | na |
-
Timing benchmarks report the mean of three runs with no measure of spread↳ Could also: Report the standard deviation, range, or coefficient of variation across the three runs alongside the mean — With only three observations, variability across runs can be substantial; reporting spread alongside the mean allows readers to judge the stability of the timing comparison and whether differences between tools exceed run-to-run noise
-
Tool comparisons are made by visual inspection of figures and MCC/time point estimates, without formal statistical tests↳ Could also: Apply a non-parametric pairwise test (e.g., Friedman test with Nemenyi post-hoc, or Wilcoxon signed-rank across dataset-size conditions) treating the multiple dataset sizes as blocks — Formal tests would provide a principled basis for distinguishing tool differences from sampling variability across dataset sizes, which is a standard approach in benchmarking studies
-
Classification quality was summarized with a single MCC value per tool-dataset combination↳ Could also: Also report precision, recall (sensitivity), specificity, and F1-score alongside MCC — MCC is a compact and balanced metric, but reporting the component metrics separately makes it easier to see whether differences stem from false-positive or false-negative behaviour, which has distinct practical implications for downstream analyses
-
The functional annotation comparison between MEDUSA and MEGAN 6 CE used a single fixed identity threshold (80%) for both tools↳ Could also: Evaluate performance across a range of identity thresholds (e.g., 70%, 80%, 90%) and report a metric such as the area under a precision-recall curve — A threshold-independent summary would show how the comparison behaves across the sensitivity-specificity trade-off space rather than at a single operating point chosen for both tools
-
Simulated datasets used a fixed random seed (--seed 5) for one dataset but genome downloads were described as random for others↳ Could also: Repeat simulations across multiple seeds and report mean ± SD of the performance metrics — Using multiple seeds would allow the results to be interpreted as estimates with uncertainty rather than as outcomes from one particular random draw of genomes, which is standard practice in simulation-based benchmarks
-
Assembly quality was assessed using MetaQUAST metrics compared against the CAMI gold-standard assembly, reported descriptively↳ Could also: Summarise assembly results with a formal metric such as NGA50 or genome fraction across complexity levels, with confidence intervals if multiple seeds are used — Standardised assembly statistics with uncertainty estimates would make it easier to compare MEGAHIT and MetaSPAdes results against published CAMI leaderboard entries using a common scale
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.