TransFlow: a Snakemake workflow for transmission analysis of Mycobacterium tuberculosis whole-genome sequencing data.
Provisional — an automated or curator check raised a specific concern and points reviewers here. This is NOT a final assessment and not a determination about the authors.
The main result did not reproduce in this reproduction attempt. Where our recomputation produced values that differ from the published ones, those discrepancies are listed below. This is a single automated attempt — not peer review and not a finding of error or misconduct — and differences can also arise from data access, undocumented parameters or the computing environment. The verdict can be contested via “report an error”.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
PARTIAL (workflow runs end-to-end; paper's real-data headline substantially reproduced). TransFlow = Snakemake MTB-transmission workflow; reproduced by running the shipped tool on (A) its 10-sample example and (B) the real 122-WGS dataset SRP058221 (P16-valid). Env fully rebuilt on «our HPC»: snakemake 7.32.4, GATK 3.8-1-0 (NO license wall), kraken1.1.1 + MiniKraken_8GB, R 4.1.2 + transcluster/adegenet/gtsummary. Gotcha solved: read-only compute-node /home -> redirect HOME/XDG_CACHE_HOME to «infra». TARGET A (example, ground truth = shipped summary_report.html): A1 MTBC-identification reproduces tightly (within-tol, max Δ0.02pp, 10/10 MTBC). A2 PANPASCO SNP-distance matrix + A3 transcluster clustering MISMATCH the shipped report (reproduced 1 cluster {S1-S6} vs expected 2 clusters; Pearson r=0.67). TARGET B (real data, paper headline B1 = 103/122 in 36 clusters @12 SNP): STRONG reproduction -> 92 (75.4%) of 122 in 35 clusters; cluster COUNT essentially exact (35 vs 36), ~11 fewer clustered samples (consistent with SNP-layer drift). 122/122 pass MTBC. ROOT CAUSE of the residual differences: the repo ships NO conda lockfile/container and the env yaml is only partially version-pinned, so a 2026-resolved environment drifts in the trimming/merge/PANPASCO layer. This is a reproducibility limitation of the published workflow, NOT evidence of fabrication (shipped values are internally plausible and the coarse real-data headline reproduces well). NOT attempted (out of scope): wet-lab VNTR genotyping/sample collection, the UKTB external validation cohort, and runtime/threads benchmarks.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-19 ⛓ 6c1c9ebca7c5
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-30
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetWGS-based transmission analysis of Mycobacterium tuberculosis requires disparate bioinformatics tools that are hard to use in a scalable, reproducible way, especially for non-experts; the authors present TransFlow to address this gap and evaluate whether it can accurately reproduce known transmission clustering/network results from a real MDR-TB WGS dataset.
- ★ TransFlow is a Snakemake- and Conda-based workflow that combines state-of-the-art tools into a single, fast, scalable pipeline for MTBC WGS-based transmission analysis. method
- ★ TransFlow comprises five coherent analysis modules: quality control, MTBC filtering, sequence alignment/variant calling, transmission detection, and transmission risk factor inference. method
- ★ Applying TransFlow to the real CTB WGS dataset reproduces transmission clustering results nearly identical to the original published study. finding
- ★ TransFlow uses a pan-genome-based approach (PANPASCO, 146 MTBC genomes covering lineages 1-4) for pairwise SNP distance calculation to avoid reference-genome lineage bias. method
- ★ TransFlow supports both SNP cutoff-based and probabilistic transmission-based (TransCluster) clustering methods to define transmission clusters. method
- ★ TransFlow reconstructs transmission networks using the SeqTrack algorithm, optionally incorporating geographic coordinates to improve local transmission inference. method
- TransFlow performs univariate regression analysis of epidemiological risk factors (e.g. age, gender, treatment history, diabetes, HIV) associated with transmission clustering using the gtsummary R package. method
- TransFlow is flexible and scalable, runnable on multi-core computing servers or personal computers with limited resources. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Whole-genome sequencing / variant calling (BWA, SAMtools, GATK, PANPASCO pan-genome alignment) | 122 MDR-TB clinical isolates (CTB study, Shanghai, China) | none | pairwise SNP distances among samples | BWA, SAMTools, GATK, PANPASCO pipeline |
| Kraken taxonomic classification / contamination screening | Paired-end FASTQ reads of MTB isolates | none | proportion of reads classified as MTBC (threshold default 90%) | Kraken with MiniKraken DB_8GB (RefSeq-based) |
| Raw read quality control and adapter trimming | Raw FASTQ sequencing reads (CTB dataset) | none | per-base quality scores, adapter content, trimming parameters | FastQC, MultiQC, Trimmomatic |
| Alignment quality control | BAM files from reference-aligned reads | none | alignment coverage and GC content across the reference genome | Qualimap2, MultiQC |
| Transmission clustering | 122 sequenced MDR-TB isolates (CTB dataset) | none | cluster assignment of samples (SNP-based and transmission-based) | R package TransCluster |
| Transmission network reconstruction | Clusters with ≥3 samples from CTB dataset | none | inferred transmission network and direction between strains | SeqTrack algorithm (R package Adegenet), GGally/ggnet2, exportable to Cytoscape |
| Univariate regression risk-factor analysis | Epidemiological metadata of CTB patients | none | association of variables (age, gender, residence, treatment history, diabetes, HIV) with transmission clustering | R package gtsummary |
| Short-read simulation for toy example dataset | Simulated MTB genome reads | none | example FASTQ/metadata for full workflow demonstration | NEAT simulator |
- – TransFlow identified 103 of 122 sequenced strains (84%) as clustered, forming 36 putative transmission clusters 103/122 (84%) in 36 clusters
- – TransFlow's clustering result closely matches the original CTB study's finding of 103 (84%) strains in 38 clusters 36 vs 38 clusters
- – Reconstructed transmission network of Cluster 2 corresponds to Cluster 9 in the original CTB study and adds supplementary putative transmission traces
- – A putative transmission scenario was reconstructed: index case (husband, 12_1614) transmitted to wife (12_0659), followed by transmission in a residential game room to three more patients, with two additional patients linked via transmission chain despite no direct epidemiological link to the game room
- – CTB analysis used transmission-based clustering with a clock rate of 1.5 SNPs/genome/year, transmission rate of 2.0, and transmission threshold of 19
- count 324 MTB isolates collected from MDR TB patients (CTB study cohort)
- count 125 samples screened by VNTR genotyping; 122 successfully sequenced by WGS (CTB sample screening)
- count 103 (84%) of 122 strains clustered into 36 transmission clusters (TransFlow transmission clustering result on CTB dataset)
- count Original CTB study: 103 (84%) in 38 clusters (Comparison to published CTB results)
- other Clock rate 1.5 SNPs/genome/year, transmission rate 2.0, transmission threshold 19 (Transmission-based clustering parameters used for CTB dataset)
- other Molecular clock rate of MTB estimated at 0.04 to 2.2 SNPs/genome/year (Reported range across MTB lineages)
- other MTBC screening threshold default 90% of reads (MTBC filtering module default parameter)
- other Pan-genome incorporating 146 MTBC complete genomes representing lineages 1-4 (PANPASCO reference pan-genome composition)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a bioinformatics applications-note describing a Snakemake pipeline (TransFlow) for Mycobacterium tuberculosis WGS transmission analysis, applied for demonstration to a retrospective observational dataset (CTB study, 122 sequenced strains). Statistical treatment centers on pairwise SNP-distance-based and probabilistic (TransCluster) clustering with fixed thresholds, network reconstruction (SeqTrack), and a univariate regression analysis of epidemiological risk factors versus transmission-cluster membership, summarized in a table with significance flagged at P<0.05. The paper reports pipeline outputs and a comparison of cluster counts to the original study rather than presenting classical hypothesis-testing results with dispersion measures or multiplicity-adjusted p-values.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Univariate (univariable) regression analysis (exact model type/family not stated) via R package gtsummary | epidemiological risk factor inference for transmission clustering (Fig. 5) | — | not stated |
-
Epidemiological risk factors were assessed one at a time using univariate regression models against transmission clustering status.↳ Could also: A multivariable regression model (e.g., multiple logistic regression) including all candidate risk factors simultaneously — This would allow each factor's association to be estimated while adjusting for the others, which can help distinguish independent associations from those driven by confounding among correlated variables.
-
Significance for multiple risk factors was judged against a P<0.05 threshold without a stated adjustment for testing several variables.↳ Could also: A multiple-comparison correction such as Benjamini-Hochberg FDR or Bonferroni — This would control the family-wise error rate or false discovery rate that can arise when several variables are each tested for association with the same outcome.
-
Transmission clusters were defined using a fixed SNP cutoff and a transmission-based threshold with specific clock-rate and transmission-rate parameters chosen based on prior literature.↳ Could also: A sensitivity analysis across a range of plausible cutoff/threshold and clock-rate values — Since molecular clock rates for MTB vary substantially between lineages, exploring a range of parameter values can show how robust the resulting clusters and network are to the specific thresholds chosen.
-
Results were reported as point summaries (e.g., percentages of clustered samples, cluster counts) without an explicitly stated measure of dispersion or uncertainty in the main text.↳ Could also: Reporting confidence intervals (e.g., for the proportion of clustered samples) or interquartile ranges/ranges for continuous variables — This would convey the precision or variability of the estimates alongside the point values.
-
The pipeline's clustering and network-reconstruction performance was illustrated by comparing its output to the previously published CTB study results on the same dataset.↳ Could also: Benchmarking against simulated transmission data with a known ground-truth network — Simulated data with known transmission links would allow direct calculation of sensitivity/specificity or accuracy metrics for the clustering and reconstruction methods, complementing comparison to a real-world prior study.
-
Transmission network reconstruction in the demonstrated workflow used a single algorithm (SeqTrack).↳ Could also: Applying and comparing multiple reconstruction algorithms (e.g., TransPhylo, Outbreaker2) on the same clusters — Since these methods rely on different modeling assumptions (e.g., handling of within-host diversity or incomplete sampling), comparing outputs across methods can provide additional confidence in the inferred transmission links.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-36469333 (TransFlow)
TransFlow is a Snakemake workflow (the paper IS a workflow/tool paper). Per BRIEF rule P16, running this tool on the paper's own data is a fully valid reproduction.
Pipeline (order of tools)
FastQC/MultiQC (QC) → Kraken1 + MiniKraken_8GB (MTBC filtering) → Trimmomatic/fastp (trim) → BWA mem → SAMtools → GATK3.8 (HaplotypeCaller/GenotypeGVCFs) variant calling against the MTB pan-genome consensus (shipped in repo) → PANPASCO (relative pairwise SNP distance) → transcluster R pkg (SNP-based clustering @ 12 SNP) → SeqTrack / adegenet (transmission network) → gtsummary (risk-factor logistic regression) → RMarkdown/knitr summary_report.html.
In scope (pipeline-derived) — TWO targets
Target A — shipped example/ data (quick, self-contained, primary)
Repo ships 10 artificial paired-end samples S1–S10 (~6.4 MB/file) + the EXPECTED
rendered output example/summary_report.html. Run the full workflow on this and
compare to the shipped report. Ground truth parsed into
original/expected_ground_truth.json:
- A1 MTBC identification % per sample (Kraken) — all 10 MTBC, 99.89–99.91 %
- A2 Pairwise SNP-distance matrix (10×10, PANPASCO relative distances)
- A3 Transmission clustering @ 12-SNP (transcluster) → 2 clusters: cluster1={S2,S3,S6,S8}, cluster2={S1,S4,S10}, unique={S5,S7,S9}; 7 clustered / 3 unique
- A4 Transmission network (SeqTrack) — structural (graded qualitatively)
- A5 Risk-factor univariable logistic regression table (gtsummary) — gt static table
Target B — paper headline on real data SRP058221 (harder, "keep going")
Paper abstract/results: "103 (84%) of 122 sequenced strains in 36 putative transmission clusters" (12-SNP cutoff). This needs the full ~122-WGS Shanghai CTB dataset (SRP058221) run end-to-end. Attempt after Target A succeeds.
Out of scope
- Wet-lab VNTR genotyping / sample collection / DNA extraction (manual, not pipeline).
- UKTB validation cohort numbers (Cluster 7: 17 isolates / 10 patients) — external validation data, not in SRP058221; out of scope.
- Runtime/threads scaling benchmarks (HW-dependent, not a reproducible value).
Compute
All on «our HPC» SLURM (partition std, no --mem). Snakemake + conda env from workflow/envs/transflow.yaml. Repo (~261 MB w/ pangenome resources) + data cloned on «infra» only. Kraken needs MiniKraken_8GB DB (downloaded on front1 to «infra»).
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.