Comparative Metagenomic Analysis of Biosynthetic Diversity across Sponge Microbiomes Highlights Metabolic Novelty, Conservation, and Diversification.
The main results reproduced, with only marginal, non-material deviations.
- ✓Same input data as the authors
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🔴Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🔴Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
R5 REPRODUCED EXACTLY. The lightest faithful in-scope result -- PERMANOVA on the PhyloFlash genus-level NTU Bray-Curtis distance across the 6 'Species extra' groups -- was rebuilt end-to-end on «our HPC» from the deposited raw reads: phyloFlash v3.4.2 (SILVA 138.1) per sample -> genus NTU table -> Bray-Curtis -> skbio PERMANOVA. Reproduced P=0.001 (pseudo-F=81.08, bacteria-only; 63.60 all-domains), matching the paper's reported P=0.001 (skbio 999-perm floor; sponge-species vs seawater microbiomes separate massively). Reproduced on 33 samples because a real DEPOSIT GAP was confirmed: the published analysis hardcodes 34 samples but ENA PRJEB51534 only contains 33 (gb8_2 / Geodia barretti Nor is missing; sw_8 deposited under alias sw_5). The upstream BGC/MAG headline numbers (R1 5082 BGCs, R2 1186 GCFs, R3 316 MAGs, R4 GTDB-Tk, R6 BiG-MAP, R7 Spearman, R8 conservation) were NOT attempted: the repo is purely downstream and NONE of the pipeline intermediates it consumes are deposited, so they cannot be regenerated without rebuilding the entire version-sensitive 420 Gbp pipeline -- this is the central reproducibility finding, not a fabrication. Deposited downstream code (DEP) verified deterministic and equivalent to the reproduction logic.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-19 ⛓ 785c3f744714
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-29
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe study investigates the global distribution and diversity of natural product biosynthetic gene clusters (BGCs) across microbial symbiont communities of different high-microbial-abundance (HMA) marine sponge species, testing whether specialized metabolic pathways (gene cluster families, GCFs) are conserved among sponge holobionts despite taxonomic and geographic differences.
- ★ The vast majority of recovered gene cluster families (GCFs) in sponge microbiomes show no similarity to any characterized BGC, revealing extreme biosynthetic novelty finding
- ★ Specialized metabolic and taxonomic profiles vary more between sponge species than within species finding
- ★ A core set of GCFs (6% shared across all species, 20% shared between at least two species) is conserved across HMA sponge holobionts, supporting an ecologically important conserved specialized metabolism finding
- ★ A widespread set of NRPS-like GCFs (elbD homolog-containing) putatively involved in production of diversified vinyl ether lipid phosphatidylethanolamine (VEPE)-related molecules is conserved within the sponge GCF core finding
- ★ GCFs encoding highly modified proteusins (RiPP class, sponge-derived RiPP/srp-like clusters) are present and appear largely unique to sponge microbiota finding
- ★ Acidobacteriota and Latescibacterota symbionts show unusually high biosynthetic potential (GCFs per MAG) compared to other phyla, marking them as potential superproducers finding
- ★ Higher bacterial taxonomic diversity does not automatically correspond to higher biosynthetic (GCF) diversity finding
- BGCs and GCFs were identified and compared using antiSMASH, BiG-SCAPE, and the MIBiG reference database, with GCF (not BGC) used as the unit of biosynthetic study to minimize fragmentation-related inflation method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| shotgun metagenomic sequencing and BGC prediction/clustering | microbiomes of sponges Geodia barretti (Norway and Canada), Aplysina aerophoba, Petrosia ficiformis, and seawater controls | none (comparative environmental survey) | BGC counts, GCF counts, gene cluster family composition and class | antiSMASH, BiG-SCAPE, MIBiG database |
| Nonpareil sequence coverage/diversity estimation | sponge and seawater metagenomes | none | estimated sequence coverage vs sequencing effort (Gbp) needed for diversity recovery | Nonpareil |
| 16S SSU rRNA gene taxonomic profiling | sponge and seawater samples | none | genus-level NTU-based Shannon alpha-diversity and prokaryotic community composition at phylum level | — |
| metagenome-assembled genome (MAG) reconstruction, binning, dereplication and taxonomic classification | sponge and seawater metagenomes | none | MAG taxonomy (GTDB r95), MAG sharedness between samples, GCF content per MAG | dRep, GTDB |
| beta-diversity and statistical community comparison (PERMANOVA, Kruskal-Wallis) | sponge and seawater samples (16S NTU and GCF content) | none | statistical significance of differences in taxonomic and GCF-based community composition between sample groups | — |
- – 5,082 BGCs detected across sponge and seawater samples, grouped into 1,186 GCFs plus 394 singletons; only 4 GCFs included MIBiG-characterized reference BGCs <1% of GCFs matched characterized BGCs
- – 65% of all sponge symbiont GCFs are unique to a single sponge species 65%
- – 58 GCFs conserved across all three sponge species; a core of 2% (16 GCFs) shared across all sample groups (including separate G. barretti geographic groups); 200 additional GCFs shared between at least two of the three species 58 GCFs (species-level core); 16 GCFs (2%, sample-group core); 200 GCFs shared pairwise
- – GCF-based Shannon alpha-diversity does not follow 16S rRNA taxonomy-based alpha-diversity across the data set Spearman r = -0.4, P = 0.01
- – PERMANOVA shows all sample groups significantly differ in both taxonomic (16S NTU) and GCF-based community composition P = 0.001
- – 316 dereplicated MAGs recovered (GTDB r95); only 3% shared between more than one sponge species 3% shared MAGs
- ▲ Acidobacteriota MAGs show the highest average GCF content among phyla (28 MAGs, avg 6.6 GCFs/MAG); Latescibacterota also highly prolific (avg 4.4 GCFs/MAG) 6.6 GCFs/MAG (Acidobacteriota); 4.4 GCFs/MAG (Latescibacterota)
- – Nine NRPS-like GCFs with conserved elbD homolog-based architecture identified within the shared GCF core, linked to ether lipid biosynthesis 9 GCFs
- count 5,082 BGCs; 1,186 GCFs; 394 singletons (total BGCs/GCFs detected across sponge and seawater metagenomes)
- correlation Spearman's r = -0.4, P = 0.01 (relationship between taxonomic (16S) alpha-diversity and GCF-based alpha-diversity)
- pvalue P = 0.001 (PERMANOVA testing of sample group differences based on 16S NTU and GCF content)
- pvalue P < 0.05 (Kruskal-Wallis pairwise differences between sponge species' taxonomy- and GCF-based alpha-diversity scores)
- mean 3.5 GCFs per sponge MAG; 2 GCFs per seawater MAG (average GCF content per recovered MAG)
- mean 6.6 GCFs per MAG (Acidobacteriota MAGs, average GCF content)
- mean 4.4 GCFs per MAG (Latescibacterota MAGs, average GCF content)
- count 316 dereplicated MAGs; 96% of 266 sponge-derived MAGs and 83% of 58 seawater-derived MAGs contained GCFs (MAG recovery and proportion encoding GCFs)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This study compared microbial taxonomic composition and biosynthetic gene cluster family (GCF) diversity across sponge species, geographic locations, and seawater using diversity statistics rather than a controlled experimental design (samples are field-collected sponge/seawater metagenomes). Alpha-diversity (Shannon index, based on 16S rRNA genus-level units and GCF content) was compared with Kruskal-Wallis tests, the relationship between taxonomic and GCF diversity was assessed with Spearman's rank correlation, and differences in community/GCF composition (beta-diversity) among sample groups were tested with PERMANOVA. Full details of these tests are stated to be reported in a supplementary table (Table S2) that is referenced but not included in the excerpted text.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Spearman's rank correlation | relationship between taxonomy-based (16S rRNA genus-level) and GCF-based Shannon alpha-diversity scores across samples | — | not stated |
| Kruskal-Wallis test | pairwise differences between sponge species' taxonomy-based and GCF-based alpha-diversity scores | — | not stated |
| PERMANOVA (permutational multivariate analysis of variance) | differences in beta-diversity among sample groups based on 16S rRNA genus-level NTU content and GCF content | — | not stated |
-
Pairwise differences in alpha-diversity between multiple sponge species/sample groups were assessed with the Kruskal-Wallis test.↳ Could also: A Kruskal-Wallis omnibus test followed by a post-hoc test such as Dunn's test with a multiple-comparison correction (e.g., Benjamini-Hochberg or Bonferroni) — This would explicitly control the family-wise or false discovery rate when many pairwise group comparisons are examined, which can be a useful addition when the number of pairwise comparisons is large.
-
The relationship between taxonomic and GCF-based alpha-diversity was tested with Spearman's rank correlation across samples.↳ Could also: A Mantel test or Procrustes analysis comparing the taxonomic and GCF dissimilarity/ordination structures — These community-level approaches can also capture multivariate concordance between two diversity data sets, complementing a single scalar diversity-index correlation.
-
Differences in community/GCF composition among sample groups were tested using PERMANOVA.↳ Could also: ANOSIM (analysis of similarities) or a distance-based redundancy analysis (db-RDA) — ANOSIM offers a rank-based alternative that is less sensitive to heterogeneity of within-group dispersion, and db-RDA could additionally allow constrained analysis of specific explanatory variables such as habitat or depth.
-
Beta-diversity results are presented via an ordination with percent explained variance on each axis, consistent with an approach such as PCoA.↳ Could also: Non-metric multidimensional scaling (NMDS) — NMDS can be preferred for compositional dissimilarity data that does not necessarily meet the assumptions of a Euclidean-based ordination, and is commonly used alongside or instead of PCoA in microbiome studies.
-
Average GCF counts per MAG (e.g., per phylum) are reported as means without an accompanying measure of dispersion in the visible text.↳ Could also: Reporting standard deviation, standard error, or interquartile range alongside these means — Adding a dispersion measure would convey the variability of GCF counts among MAGs within each phylum, which can be informative given the differing numbers of MAGs per phylum.
-
MAG sharedness/similarity across samples was determined using dRep clustering with a genome average nucleotide identity (gANI) threshold (<95%) rather than a formal statistical test.↳ Could also: A permutation-based test of MAG/strain sharing frequency against a null distribution — This could complement the descriptive clustering-based sharedness metric with a formal significance assessment of whether observed sharing patterns exceed what would be expected by chance.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-35862823
Paper: Loureiro et al. 2022, mSystems. "Comparative Metagenomic Analysis of Biosynthetic Diversity across Sponge Microbiomes Highlights Metabolic Novelty, Conservation, and Diversification." DOI 10.1128/msystems.00357-22 · PMCID PMC9426513.
Repo: https://github.com/CatarinaCarolina/sponge_meta_BGC (default branch master,
last pushed 2023-04-21). Data: ENA PRJEB51534.
1. What kind of study this is
A shotgun-metagenomics comparative study of biosynthetic gene clusters (BGCs) across the microbiomes of three sponge species (+ seawater controls). The full analytical pipeline is:
raw WGS reads (ENA PRJEB51534, 420 Gbp)
→ QC/trimming
→ metagenome assembly (per sample)
→ genome binning → dRep dereplication → 316 MAGs
→ GTDB-Tk taxonomy
→ antiSMASH v5.0 (BGC detection) → 5,082 BGCs
→ BiG-SCAPE (BGC → GCF clustering) → 1,186 GCFs + 394 singletons
→ BiG-MAP (BGC abundance/RPKM across samples)
→ PhyloFlash (16S/SSU NTU taxonomic profiling)
→ downstream Python plotting/stats scripts (THE DEPOSITED REPO)
→ Figures 1–4 + diversity statistics
2. What is actually deposited vs. needed
| Artifact | Needed by | Deposited? | Where |
|---|---|---|---|
| Raw WGS reads | everything | YES | ENA PRJEB51534 (39 runs / 33 samples) |
| Sample metadata | all scripts | YES | repo metadata/sub_sponge_metadata.csv, ..._extra.xls |
| Downstream plotting/stats scripts | Figs 1–4 | YES | repo (17 .py scripts) |
Network_Annotations_Full.tsv (antiSMASH/BiG-SCAPE) |
Figs 1,2,4 | NO | — must regenerate |
mix_c0.50.network, mix_clustering_c0.50.tsv (BiG-SCAPE) |
Figs 1,2,4 | NO | — must regenerate |
phyloFlash_compare.6.ntu_table.tsv (PhyloFlash) |
Fig 3 taxa | NO | — must regenerate |
all_RPKMs_norm.tsv (BiG-MAP) |
Fig 3 BGC | NO | — must regenerate |
Cdb.csv (dRep clustering) |
Fig 4 | NO | — must regenerate |
gtdbtk.bac120.summary.tsv (GTDB-Tk) |
Fig 4 | NO | — must regenerate |
all_refined_bins/, representative_genomes.txt |
Fig 4 | NO | — must regenerate |
Critical reproducibility gap: the repo is purely downstream. Every one of its scripts consumes an intermediate data product of the heavy pipeline, and none of those intermediates are deposited anywhere (not in the repo, not in the paper supplement — Tables S1–S6 are metadata/QC/diversity-stats/curated-example tables, not the BiG-SCAPE network or antiSMASH annotation files; Data Availability lists only ENA + GitHub). Therefore no figure or reported count can be regenerated by simply running the deposited code — one must first reconstruct the entire upstream pipeline from the 420 Gbp of raw reads.
3. In scope vs out of scope
In scope (pipeline-derived, in principle reproducible):
- R1 Total BGCs detected by antiSMASH v5 (reported 5,082).
- R2 GCFs from BiG-SCAPE at c0.50 (reported 1,186 GCFs + 394 singletons).
- R3 Dereplicated MAG count from dRep (reported 316; 266 sponge + 58 seawater).
- R4 GTDB-Tk taxonomy of MAGs (phylum-level classification of all 316).
- R5 PhyloFlash NTU taxonomic diversity → Bray-Curtis → PERMANOVA P=0.001.
- R6 BiG-MAP BGC RPKM diversity → ordination/Shannon (Fig 3).
- R7 Spearman correlation taxonomy vs BGC alpha-diversity (reported r=-0.4, P=0.01;
the script computes
spearmanr, paper text says "Spearman"). - R8 Conservation/sharing counts (GCFs shared across species; 58 conserved; etc.).
- DEPOSITED-CODE CHECK the 17 downstream scripts are deterministic
(pandas/scipy/sklearn
spearmanr,kruskal,PERMANOVA,PCoA) — given genuine intermediates they reproduce R5–R8 exactly. Auditable independently of the pipeline.
Out of scope (not pipeline-derived / manual / external):
- MIBiG reference-BGC curation, AdenylPred A-domain substrate predictions (Table S5), manual GCF/RiPP family inspection, th
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.