Deep learning from phylogenies to uncover the epidemiological dynamics of outbreaks.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No relevant deviation in data/preprocessing
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- ✓Overall, the reproduction was clean
- Every checked point held up.
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
REPRODUCED (1:1). phylodeep is a PyPI package bundling the paper's pretrained CNN-CBLV & FFNN-SS estimators for BD/BDEI/BDSS birth-death models; applying it to the shipped 200-tip HIV Zurich tree is a deterministic forward pass, the cleanest 1:1 target. Built a python-3.9 conda env on «infra» (mamba; front-node /tmp full + /home over quota + «infra» reports 0 inodes which breaks plain conda -- mamba sidesteps the check), installed phylodeep 0.9 / TF 2.17, cloned repo @02fd0e6, and ran modeldeep + paramdeep(both encodings, +CI) + checkdeep + an FFNN-SS robustness sweep via SLURM «job» on node n093. Every reported Fig-5 value reproduces to the printed precision: model selection BDSS prob 1.00; R0 1.60(FFNN)/1.69(CNN); infectious period 10.2/9.8 yr; X_ss ~9.3; f_ss ~0.072-0.079; and the robustness R0 sweep 1.54/1.60/1.66 to the digit. CI bounds match the repo README exactly. Only the visual adequacy plot is 'partial' (reproduced, not auto-compared). Did NOT attempt the heavy simulated-tree accuracy curves (Figs 2-3) or from-scratch training (out of scope).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 94assessed: 2026-06-21 ⛓ 2791167129b1
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-21
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-21no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThat a likelihood-free, simulation-based deep learning approach—trained on millions of simulated phylogenies represented either as summary statistics or as a compact vectorial encoding—can perform epidemiological parameter estimation and model selection in phylodynamics more accurately and scalably than standard likelihood-based/Bayesian methods.
- ★ Deep learning (FFNN-SS and CNN-CBLV) enables accurate and fast likelihood-free estimation of epidemiological parameters and model selection from phylogenies finding
- ★ CBLV is a novel compact, bijective vectorial representation of phylogenetic trees that preserves full topology and branch-length information method
- ★ Deep learning methods are more accurate than BEAST2 for BDEI and BDSS models, and comparable for the simpler BD model finding
- ★ The deep learning approach scales to very large phylogenies (thousands of tips) by averaging predictions across disjoint subtrees method
- ★ BEAST2 shows non-convergence and convergence to local optima more frequently for complex models (BDEI, BDSS) due to numerical ODE approximations finding
- TreePar maximum-likelihood estimation exhibited overflow/underflow issues and produced inaccurate model comparisons for complex models finding
- ★ The method was applied to an HIV dataset from the MSM community in Zurich to assess superspreading dynamics finding
- ★ PhyloDeep software implementing these methods is publicly available (GitHub, PyPI, Docker Hub) resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| simulation-based deep learning inference (FFNN-SS, CNN-CBLV) | simulated phylogenies under BD, BDEI, BDSS birth-death models | none (parameters varied: R0, infectious period, incubation period, fSS, XSS) | estimated epidemiological parameters and selected model class | PhyloDeep (custom FFNN/CNN) |
| Bayesian phylogenetic inference | same simulated test trees (100 trees, 200-500 tips) | none | posterior (median a posteriori) parameter estimates | BEAST2 |
| maximum-likelihood estimation | simulated test trees | none | point parameter estimates and model likelihoods | TreePar |
| deep learning phylodynamic inference on decomposed subtrees | simulated 'huge' trees (5000-10,000 tips) | none | averaged parameter estimates across subtrees | PhyloDeep |
| phylodynamic analysis of real-world sequence data | HIV-1 sequences from men-having-sex-with-men (MSM) community, Zurich | none | superspreading dynamics parameters (fSS, XSS, R0) | PhyloDeep |
- ▲ Deep learning methods (FFNN-SS, CNN-CBLV) were more accurate than BEAST2 for BDEI and BDSS models
- – Deep learning methods were comparable/near-optimal to BEAST2 for the BD model
- – BEAST2 failed to converge in a subset of cases 2% of BDEI cases, 15% of BDSS cases
- – BEAST2 likely converged to local optima, indicated by high relative error (>1.0) estimates 8% of BDEI cases, 11% of BDSS cases
- ▲ Deep learning accuracy improved with larger tree size
- – TreePar MLE showed overflow/underflow (infinite likelihood values) and inaccurate model comparisons for complex models, more prominent for larger trees
- count 4 million simulated trees per model (training data generated for each phylodynamic model (BD, BDEI, BDSS))
- count 3.99 million training trees (size of neural network training set)
- count 10,000 validation trees (used for early stopping during training)
- count 10,000 testing trees (100 also evaluated with BEAST2/TreePar) (test set for accuracy comparison against gold-standard methods)
- count 1 million trees (used to define confidence intervals for estimated parameters)
- count 10,000 huge trees, 5000-10,000 tips each (evaluating applicability to very large phylogenies)
- other 2% (BDEI) / 15% (BDSS) (BEAST2 non-convergence rate by model)
- other 8% (BDEI) / 11% (BDSS) (BEAST2 cases with relative error >1.0, suggesting convergence to local optima)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a simulation-based, computational methods paper (not a wet-lab study) that develops deep-learning approaches (FFNN on summary statistics; CNN on a compact vector tree representation) for phylodynamic parameter estimation and model selection, and benchmarks their accuracy against the established tools BEAST2 and TreePar. Accuracy was assessed by simulating millions of phylogenetic trees under three birth-death models, holding out a 100-tree test set, computing a per-tree 'relative error' between estimated and true parameter values, and comparing methods' relative-error distributions with a two-sided paired z-test per parameter/model combination. Results are reported mainly as average relative error, per-tree scatter plots, and a binary significance indicator (P<0.05 shown as a line, non-significant omitted), plus confidence intervals for parameter estimates derived from a separate 1-million-tree simulation set.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| two-sided paired z-test | comparing relative-error accuracy of BEAST2 vs FFNN-SS vs CNN-CBLV, per estimated parameter, separately for BD-, BDEI-, and BDSS-simulated trees (Fig. 3) | 100 test trees | not stated |
-
Accuracy of the three methods (BEAST2, FFNN-SS, CNN-CBLV) was compared separately for each parameter and model using two-sided paired z-tests, shown as a binary significant/non-significant indicator per comparison.↳ Could also: A repeated-measures ANOVA or Friedman test across the three methods, followed by corrected post-hoc pairwise comparisons — This would provide a single omnibus test of whether the methods differ before conducting pairwise follow-ups, which is a standard way to structure comparisons among more than two related methods on the same set of test trees.
-
Multiple paired z-tests were performed across several parameters and three separate models without a stated adjustment for multiple comparisons.↳ Could also: A multiplicity correction such as Benjamini-Hochberg FDR or Bonferroni across the full family of z-tests — Correcting for the number of tests performed across parameters and models is a standard approach to control the overall false-positive rate when many related significance tests are reported together.
-
Statistical significance for method comparisons is reported as a threshold indicator (P<0.05 shown as a line; non-significant results omitted) rather than as exact p-values.↳ Could also: Reporting exact/continuous p-values alongside the significance calls — Exact p-values let readers gauge the strength of evidence on a continuous scale rather than relying solely on a binary cutoff.
-
Method comparisons rely on relative error and a significance test but do not report a standardized effect size for the magnitude of accuracy differences.↳ Could also: A standardized effect size (e.g., Cohen's d for the paired differences, or a rank-based effect size such as Cliff's delta) — An effect size complements a significance test by quantifying how large the accuracy difference between methods is, independent of sample size.
-
The paired z-test was used to compare relative-error distributions between methods on the 100-tree test set.↳ Could also: A non-parametric alternative such as the Wilcoxon signed-rank test, or bootstrap/permutation-based confidence intervals for the paired differences — Relative-error metrics can be skewed or bounded away from normal, so a distribution-free or resampling-based test can serve as a complementary check that does not rely on the z-test's normality assumption.
-
Confidence intervals for the estimated epidemiological parameters were derived from an additional, separately simulated set of 1 million trees rather than from the same test-set replicates used for the accuracy comparison.↳ Could also: Bootstrap resampling of the existing test-set estimates to construct confidence intervals — Deriving intervals directly from the evaluated sample (via bootstrap) is a standard complementary approach that ties uncertainty estimates to the same data used for the reported accuracy comparison.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — PMID 35794110 (phylodeep)
Paper: Voznica et al. 2022, Nat Commun 13:3896. "Deep learning from phylogenies to uncover the epidemiological dynamics of outbreaks." DOI 10.1038/s41467-022-31511-0 · PMCID PMC9258765.
Code: https://github.com/evolbioinfo/phylodeep (default branch main,
HEAD 02fd0e6da116d2db50ce7869488f57902141b157, pushed 2024-10-09; tag 0.9
= b03f3c4). PyPI package phylodeep (latest 0.9). The package bundles the
paper's pretrained neural networks (CNN-CBLV and FFNN-SS for BD/BDEI/BDSS) and
exposes 3 CLI tools: checkdeep (a priori adequacy), modeldeep (model
selection), paramdeep (parameter inference). License: NOASSERTION (GPL-style
text in repo).
Data:
- Zenodo
10.5281/zenodo.6646668— the full simulated training/testing sets (50k-tree banks) + encoded representations used to train/evaluate the networks. test_tree_HIV_Zurich/Zurich.trees(in repo) — the real HIV-1 MSM phylogeny from Zurich (~200 tips), the paper's showcase application. Ultimately derived from Rasmussen et al. 2017 / Swiss HIV Cohort (the underlying sequences are access-restricted, but the time-scaled tree itself ships in the repo).
What the paper reports (pipeline-derived) and our scope decision
| Result block | Pipeline | In scope? | Why |
|---|---|---|---|
| HIV application (Fig 5): model selection picks BDSS (prob 1.00); paramdeep estimates R0≈1.6–1.7, infectious period≈9.8–10.2 yr, superspreading ratio X_ss≈9, superspreader fraction f_ss≈7–8%; checkdeep adequacy | run pretrained CNN-CBLV + FFNN-SS on the shipped Zurich tree | YES — PRIMARY | Deterministic forward pass through bundled pretrained nets on a shipped real tree. The single cleanest 1:1 target: same tree + same weights ⇒ should reproduce exactly. |
| Accuracy of NN estimators on 10k held-out simulated trees (Figs 2–3, relative error / CI coverage vs BEAST2 & true params) | encode simulated test trees → NN predict → compare to ground truth | PARTIAL / STRETCH | Reproducible in principle from the Zenodo testing banks + bundled nets, but heavy (large banks, BEAST2 gold-standard reruns). Attempt after the primary if feasible; the BEAST2 comparison itself is a separate gold-standard rerun (very heavy) and likely out. |
Network training from scratch (Fig 1 architecture, simulators/) |
TreeSimGM/treesimulator simulations + Keras training | OUT | Training the published nets from scratch is not a "reproduce the reported number" task and is enormous; the paper's claim is the trained estimator's behaviour, which we test via the bundled weights. |
| Wet-lab / epidemiological interpretation, contact-network biology | none (domain interpretation) | OUT | Not pipeline-derived. |
Reproduction plan (primary)
- Front node (internet):
pip install phylodeepinto a conda env on «infra»; this pulls the bundled pretrained models. Clone repo on «infra» for the tree. - Sampling probability per paper: p ≈ 0.25 (0.45 × 0.56; robustness band
0.20–0.30). Tree:
Zurich.trees(~200 tips → uses the full-tree FFNN-SS + CNN-CBLV models, which cover 200–500 tips). - SLURM job (compute-only): run
modeldeep(model selection),paramdeep -m BDSSwith both encodings (-v SUMSTATS→ FFNN-SS,-v CBLV→ CNN-CBLV) and-cfor CIs, andcheckdeep. Capture CSV outputs. - Compare reproduced point estimates to Fig-5 reported values →
agreement.json. Also profile both datasets in the same pass →data/dataset_profile.json.
Honesty notes
- Fig-5 values are read off a figure, so reported numbers are approximate (point estimates ± CI). Grading uses within-tol on the figure-read values; the decisive check is the model-selection call (BDSS prob 1.00) and the order-of-magnitude/point estimates.
- Package version deviation (0.9 vs the ~0.2.x contemporary with the 2022 paper) is recorded; the weights are the paper's and are stable across versions, so predictions should be identic
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
This is a clean 1:1 reproduction. The authors ship both the pretrained phylodeep estimators (PyPI/Zenodo) and the exact 200-tip HIV Zurich phylogeny, so the Fig-5 application is a deterministic forward pass that reproduces every reported value to printed precision: BDSS model selection prob 1.00, R0 1.60/1.69, infectious period 10.2/9.8 yr, X_ss ~9, f_ss ~0.072-0.079, plus the sampling-probability robustness sweep (1.54/1.60/1.66) to the digit, with CIs matching the repo README. No deviation sits on the authors' side, our method, or data availability. The sole non-exact item (C8 checkdeep adequacy plot) was reproduced but not auto-compared visually — out-of-scope tooling, not a discrepancy.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.