hgtseq: A Standard Pipeline to Study Horizontal Gene Transfer.
The main results reproduced, with only marginal, non-material deviations.
- ✓Same input data as the authors
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
IN PROGRESS. Paper = nf-core/hgtseq (enrichment code_url 'marcelm/cutadapt' is wrong; real repo github.com/nf-core/hgtseq, MIT, release 1.0.0 commit 4cd916f contemporaneous with paper). Data PRJNA785331 = 9 human WXS exomes, verified 9/9 on ENA. Dataset profiled. Reproduction plan tiered: T1 run -profile test (pipeline runs e2e), T2 run 9 COPD exomes vs GRCh38+Kraken2 std DB to reproduce microbial-proportion claims C3/C4 (2.44e-6, 1.04e-5; DB-version sensitive), T3 stretch T2T-vs-GRCh38 deltas. Performance metrics (154 CPU-hrs etc.) marked OUT OF SCOPE (environment-specific, not 1:1 reproducible). BLOCKER: «our HPC» VPN tunnel down at start (ssh timeout to «host».«infra») — waiting for central fix, retrying. Not a drop; control-plane work complete.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 63assessed: 2026-06-19 ⛓ 46f90d951de9
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-29
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator headless) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetHorizontal gene transfer (HGT) from microbes is known in prokaryotes but poorly characterized in mammals/humans, and no highly automated, standardized pipeline exists to systematically detect candidate HGT signatures from unmapped sequencing reads; the paper tests whether such a pipeline (hgtseq) can reliably detect and quantify microbial-derived sequences across diverse mammalian exome datasets.
- ★ hgtseq is a fully automated, portable, and scalable Nextflow/nf-core pipeline for detecting horizontal gene transfer signatures from unmapped sequencing reads. method
- ★ Few existing tools for investigating HGT are highly automated or scale across raw-data-to-results workflows in any computing environment. finding
- ★ hgtseq accepts either raw FASTQ paired-end reads or pre-aligned BAM files and can run seamlessly on-premise (Slurm HPC) or on the cloud (AWS Batch). method
- ★ Analysis of six exome datasets from five mammals (human x2, Bos taurus, Mus musculus, Canis lupus familiaris, Macaca fascicularis) detected microbial-classified reads among unmapped reads in every dataset, consistent with potential HGT signatures. finding
- ★ Contaminant genera (from DNA extraction kits/sample handling) confound HGT detection, requiring a curated flagged/blacklisted genera list to interpret classified reads. method
- Using the T2T-CHM13 human reference instead of GRCh38 modestly reduces the number of unmapped and microbial-classified reads in the COPD dataset. finding
- hgtseq was built from 21 shared nf-core modules and 4 local modules across 329 commits, released as version 1.0.0. resource
- hgtseq's COPD exome runtime (~3h) is comparable to or faster than previously reported HGT-detection tool runtimes (4.3-23.4h), though direct comparison is difficult. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| whole exome sequencing + Kraken2 taxonomic classification of unmapped reads | human (COPD dataset) | none | proportion of unmapped reads classified to microbial genera; runtime/resource usage | Kraken2, BWA-MEM, Qualimap, MultiQC |
| whole exome sequencing + Kraken2 taxonomic classification of unmapped reads | human (CHIKV dataset) | none | proportion of unmapped reads classified to microbial genera | Kraken2 |
| whole exome sequencing + Kraken2 taxonomic classification of unmapped reads | Bos taurus | none | proportion of unmapped reads classified to microbial genera | Kraken2 |
| whole exome sequencing + Kraken2 taxonomic classification of unmapped reads | Canis lupus familiaris | none | proportion of unmapped reads classified to microbial genera | Kraken2 |
| whole exome sequencing + Kraken2 taxonomic classification of unmapped reads | Macaca fascicularis | none | proportion of unmapped reads classified to microbial genera | Kraken2 |
| whole exome sequencing + Kraken2 taxonomic classification of unmapped reads | Mus musculus | none | proportion of unmapped reads classified to microbial genera | Kraken2 |
| taxonomic composition visualization (Krona) of single-unmapped classified reads after blacklist filtering | human, Bos taurus, Mus musculus, Canis lupus familiaris, Macaca fascicularis (cross-species comparison) | none | qualitative similarity/difference in microbial genus composition between species | Krona |
| reference genome comparison (read alignment/classification pipeline) | human (COPD dataset) | reference genome swap: T2T-CHM13 vs GRCh38 | change in total unmapped reads and microbial-classified reads | hgtseq pipeline (BWA-MEM, Kraken2) |
- – hgtseq completed full analysis of a human exome (COPD dataset) in about 3 hours, generating 150 jobs, 154.3 CPU/hours, and 1136.87 GB memory usage. ~3 h; 154.3 CPU/hours
- – Human datasets showed proportions of unmapped reads classified to microbial genera ranging from 1.04x10^-5 (CHIKV, both mates unmapped) to 2.44x10^-6 (COPD, both mates unmapped). 1.04e-5 to 2.44e-6
- – Bos taurus showed proportions of classified microbial reads ranging from 8.5x10^-5 to 8.67x10^-5 across the two unmapped-read categories. 8.5e-5 to 8.67e-5
- – Canis lupus familiaris showed 3.42x10^-8 (both mates unmapped) and 7.32x10^-7 (single mate unmapped) proportions of classified microbial reads. 3.42e-8 to 7.32e-7
- – Macaca fascicularis and Mus musculus both showed detectable proportions of classified microbial reads (1.16e-5/2.14e-5 and 4.91e-6/6.21e-6 respectively). 1.16e-5 to 6.21e-6 range
- – Proportion of contaminant-flagged genera was generally higher among reads with lower Kraken2 classification scores in most datasets, but not in the human COPD or Bos taurus datasets.
- ▼ Using T2T-CHM13 instead of GRCh38 for the human COPD dataset reduced total unmapped reads (-1.13% single-unmapped, -0.91% both-unmapped) and microbial-classified reads (-1.46% single-unmapped). -0.91% to -1.46%
- – hgtseq pipeline comprised 21 shared nf-core modules and 4 local modules built over 329 commits, released as version 1.0.0. 329 commits; 25 modules
- count 329 commits (total commits to the hgtseq pipeline repository excluding shared repo contributions)
- count 21 shared nf-core modules + 4 local modules (composition of the hgtseq pipeline)
- other 154.3 CPU/hours; 1136.87 GB memory; 150 jobs; ~3 h runtime (resource usage for full pipeline run on human COPD exome dataset)
- other 3742.28 GB read, 2389.91 GB written (total I/O for COPD dataset pipeline run)
- fold_change 1.04x10^-5 to 2.44x10^-6 (proportion of unmapped reads classified to microbial genera in human datasets (both mates unmapped category))
- count 49,516,660 to 198,822,100 reads (range of sample read counts across the six tested datasets (Bos taurus lowest, Canis lupus familiaris highest))
- fold_change -1.13% (single-unmapped), -0.91% (both-unmapped), -1.46% (microbial classified, single-unmapped) (change in read counts when using T2T-CHM13 vs GRCh38 reference for human COPD dataset)
- other 4.3 h (HGT-ID) to 23.4 h (VirusFinder2) (reported runtimes of previously developed HGT/viral-integration detection tools, cited for comparison)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper describes a bioinformatic pipeline (hgtseq) for detecting microbial sequence signatures in unmapped reads from host exome/genome sequencing data, and demonstrates its use on six exome datasets from five mammalian species. Results are reported primarily as descriptive proportions (e.g., proportion of unmapped reads classified to microbial genera relative to total reads), performance/benchmarking metrics (runtime, CPU/memory usage), and qualitative visual comparisons of Krona plots and stratified score distributions across datasets. No formal inferential statistical hypothesis tests, p-values, or replicate-based variability estimates are reported in the text provided.
-
Differences in microbial-read proportions across species/datasets (e.g., 1.04 × 10⁻⁵ in human CHIKV vs 3.42 × 10⁻⁸ in Canis lupus familiaris) are reported as single point estimates without a formal statistical comparison.↳ Could also: A test for comparing proportions/rates, such as a chi-square test, Fisher's exact test, or a Poisson/negative-binomial regression on read counts — These approaches would allow a formal assessment of whether observed differences in classified-read proportions across datasets exceed what could be expected by chance, complementing the descriptive proportions already presented.
-
Comparisons between datasets (e.g., proportion of flagged/blacklisted contaminant genera by classification score, or Krona plot composition across species) are described through qualitative visual inspection.↳ Could also: Quantitative similarity/diversity metrics (e.g., Bray-Curtis dissimilarity, Jensen-Shannon divergence) with associated statistical testing (e.g., PERMANOVA) commonly used in microbiome composition comparisons — Such metrics would provide a numerical, reproducible measure of compositional similarity or difference between datasets, complementing the qualitative visual comparison of Krona plots.
-
Each species/dataset in the demonstration appears to be represented by a single sample or cohort-level aggregate, without stated biological replicates.↳ Could also: Including multiple biological replicates per species/condition and reporting a dispersion measure (e.g., SD or 95% CI) around the proportion estimates — Replicate samples would allow variability in microbial-read proportions to be quantified and would support formal inferential comparisons between species or conditions, in addition to the single-sample point estimates presented.
-
The comparison between GRCh38 and T2T-CHM13 references for the human COPD dataset reports percentage changes in unmapped/classified read counts without a formal statistical test.↳ Could also: A paired comparison test (e.g., paired t-test or Wilcoxon signed-rank test) if multiple samples were run against both references — A paired test could formally quantify whether the reference-related shift in read classification is consistent and statistically distinguishable from expected variability, if additional samples were available.
-
Pipeline performance (runtime, CPU/memory usage) is compared descriptively to previously published tools (e.g., HGT-ID at 4.3 h vs VirusFinder2 at 23.4 h), with the text noting differences in methods and environments.↳ Could also: A standardized benchmarking framework with repeated runs and reported variability (e.g., mean ± SD runtime across replicate runs on matched hardware) — Repeated-run benchmarking with variability estimates would let performance comparisons account for run-to-run variation, complementing the single-run descriptive figures already reported.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-36498841 (hgtseq: A Standard Pipeline to Study Horizontal Gene Transfer)
- Paper: Carpanzano, Santorsola, nf-core community, Lescai. Int J Mol Sci 2022. DOI 10.3390/ijms232314512. PMCID PMC9738810.
- Real pipeline repo (the enrichment
marcelm/cutadaptis WRONG — cutadapt is just one tool used inside): https://github.com/nf-core/hgtseq — MIT, public, not archived.- Release 1.0.0 (2022-10-24, commit
4cd916f0c15e2c1a6a4a09228981cfd919b33222) is contemporaneous with the paper (Zenodo DOI 10.5281/zenodo.7244735). This is the version pinned for reproduction. - This is an nf-core Nextflow DSL2 pipeline → highly reproducible by design (containerised, versioned). Per BRIEF rule P16, running this published tool on the paper's data is fully valid.
- Release 1.0.0 (2022-10-24, commit
- Data: SRA/ENA PRJNA785331 — 9 human whole-exome (WXS) paired-end runs (SRR17085822–SRR17085830), Illumina HiSeq X Ten, COPD/Chinese cohort. Open access. Profiled in
data/dataset_profile.json(N reported 9 == N observed 9).
Pipeline (what hgtseq does)
TrimGalore (cutadapt+FastQC) → BWA-mem align to host genome (GRCh38 or T2T-CHM13) → samtools splits unmapped reads into both-mates-unmapped and single-mate-unmapped → Kraken2 taxonomic classification of those reads → Krona viz → RMarkdown report computing per-category microbial-read proportions + candidate integration loci. QC via FastQC/Qualimap/Bamtools/MultiQC.
IN SCOPE (pipeline-derived, attempt to reproduce)
| id | what | reported | paper loc | reproducibility notes |
|---|---|---|---|---|
| C1 | dataset N | 9 human WXS samples | Results/Methods | structural — already verified (9/9 on ENA) |
| C2 | pipeline runs end-to-end & emits documented outputs | qualitative | repo/paper | -profile test on «our HPC» (canonical nf-core repro check) — deterministic |
| C3 | microbial-seq proportion, COPD, both mates unmapped | 2.44e-6 | Fig 3 / Table S1 | needs full data + GRCh38 + Kraken2 std DB; exact value DB-version-sensitive |
| C4 | microbial-seq proportion, COPD, single mate unmapped | 1.04e-5 | Fig 3 / Table S1 | same as C3 |
| C5 | T2T-CHM13 vs GRCh38: unmapped-read reduction | −1.13% (single), −0.91% (both) | Results | requires running both references; harder/optional |
| C6 | T2T vs GRCh38: microbial-reads reduction (single-unmapped) | −1.46% | Results | optional stretch |
OUT OF SCOPE (cannot be reproduced 1:1 — honest exclusion)
- Performance metrics — "~3 h", "150 jobs", 154.3 CPU/hours, 1136.87 GB memory, I/O 3742.28 GB read / 2389.91 GB written (Results). These are environment-specific (cluster, scheduler, container engine, parallelism) and are NOT a property of the data/algorithm; reproducing them on «our HPC»/SLURM is meaningless as a 1:1 comparison. Recorded as reported, explicitly not attempted.
- Any wet-lab / manual interpretation of specific candidate HGT loci (biological judgement, not pipeline output).
Reproduction plan (tiered; 80% floor then push)
- Tier 1 (quick minimum, C2):
nextflow run nf-core/hgtseq -r 1.0.0 -profile test,singularityon «our HPC» front1 (nodes have internet) / SLURM. Confirms the published pipeline executes and produces the documented output tree. Deterministic smoke test. - Tier 2 (the real claim, C3/C4): download the 9 COPD exomes to «infra», run full hgtseq with GRCh38 + Kraken2 standard DB (k2_standard, as referenced by the pipeline's own test_full), compute both microbial-proportion metrics, compare to 2.44e-6 / 1.04e-5. Flag DB-version sensitivity.
- Tier 3 (stretch, C5/C6): re-run vs T2T-CHM13 to reproduce the reduction percentages.
All heavy compute on «our HPC» via SLURM; all data on «infra». Blocker as of start: «our HPC» VPN tunnel down — waiting for central fix, retrying periodically.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.