GenTB: A user-friendly genome-based predictor for tuberculosis resistance powered by machine learning.
The main results reproduced: recomputed values matched the published ones within tolerance.
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough; both shipped GenTB predictors reproduced as DETERMINISTIC pipeline checks. R3 (GenTB-RF): the shipped TBpredict.R + 13 randomForest 4.6-14 models on the shipped 3-strain example matrix regenerate the shipped expected matrix.json EXACTLY (39/39 strain x drug cells: resistance prob + OOB FN/FP; top-5 'important' mutations also identical; only an auxiliary 'other' display field differs due to R duplicate-column-name handling). R1 (GenTB-WDNN): the shipped Keras-1.2.0 frozen weights applied to the shipped 792-isolate validation set give per-drug ROC-AUCs that track Chen 2019 EBioMedicine Table 2 within ~0.035 for 7/10 drugs (rif,inh,emb,str,amk,oflx,kan) and within ~0.01-0.02 in aggregate (first-line 0.921 vs 0.937, second-line 0.900 vs 0.891); 3 small-positive-class/hard drugs deviate more (pza 0.835/0.944, cap 0.803/0.872, moxi 0.950/0.859) - expected because the reference Table 2 uses a RETRAINED WDNN whereas this uses the shipped frozen weights. Root-caused + fixed the two prior requeues (env-builds died on a read-only compute-node conda cache; plus a set -u / unbound $ADDR2LINE conda-activation abort). NOT attempted: the 19,880-isolate benchmark (matrix not deposited) and the non-deterministic CV retrain. The authors' pinned Keras1.2/TF1.6 stack builds but fails to construct the Merge layer on modern TF (concat arg order) - inference done via an exact numpy forward pass. Brief accession zenodo:883859 is wrong (= seaborn).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-25
- Rubric version
- not recorded
- Assessed by
- —
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetCan a free, user-friendly web-based application (GenTB) using machine learning (Random Forest and Wide-and-Deep Neural Network classifiers) predict Mycobacterium tuberculosis drug resistance from whole-genome/targeted sequencing data with accuracy comparable to or better than existing leading resistance-prediction tools (Mykrobe, TB-Profiler)?
- ★ GenTB is a free, open, web-based application offering two ML predictors (Random Forest and WDNN) that predict resistance to 13 and 10 anti-TB drugs, respectively. resource
- ★ All four tools (GenTB-RF, GenTB-WDNN, TB-Profiler, Mykrobe) reliably predicted resistance to first-line TB drugs but showed varying performance for second-line drugs. finding
- ★ GenTB-RF and GenTB-WDNN had marginally higher mean sensitivity across nine shared drugs than TB-Profiler and Mykrobe, at the expense of up to 1.5% lower specificity. finding
- ★ Genotypic resistance sensitivity for isoniazid and rifampicin was substantially lower on isolates sequenced at low depth (<10x across 95% of the genome), highlighting the need for input sequence quality control. finding
- The GenTB raw-read processing pipeline uses fastQValidator, fastp, Kraken, minimap2, Samtools, and Pilon for quality control, alignment, and variant calling against the H37Rv reference. method
- The pyrazinamide RF model was retrained on variants in pncA, panD, clpC1, and clpP genes using a larger, more recent isolate dataset. method
- GenTB-WDNN is a multitask logistic regression model combined with a multilayer perceptron, previously shown to perform equal to or better than RF architectures trained on the same data. mechanism
- A user questionnaire among past GenTB users indicated the application is clear, pleasing, stable, easy to use, and usable. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Whole-genome/targeted sequencing-based genotypic resistance prediction benchmarked against phenotypic drug-susceptibility testing | 20,408 clinical M. tuberculosis Illumina sequence isolates | none | sensitivity, specificity, AUC per drug per tool | Illumina sequencing; GenTB, TB-Profiler 2.8.12, Mykrobe v0.9.0 |
| Targeted sequencing at 18 drug-resistance loci via molecular inversion probes plus binary culture-based DST | 1,397 clinical M. tuberculosis isolates | none | SNPs/indels used to build Random Forest classifiers for 13 drugs | molecular inversion probes; randomForest R package v4.6.7 |
| Retraining of pyrazinamide resistance Random Forest model on WGS variant data | 20,365 M. tuberculosis isolates (15,267 training, 5,098 validation) | none | pyrazinamide resistance classification from 393 variants in pncA, panD, clpC1, clpP | randomForest R package v4.6-14 |
| Wide and Deep Neural Network training on WGS-derived variant features | 3,601 M. tuberculosis isolates | none | resistance prediction to 11 drugs from 222 SNP/indel features | Keras 2.2.4 / TensorFlow 1.8.0 (Python 3.6) |
| Sequencing depth analysis of resistance loci | M. tuberculosis clinical isolates from validation dataset | low vs adequate sequencing depth (<10x vs higher across 95% of genome) | coverage depth and its effect on sensitivity of resistance prediction | Mosdepth v0.2.9 |
| User experience questionnaire | 166 registered GenTB users (12 respondents) | none | Likert-scale ratings (0-10) of clarity, pleasantness, ease of use, stability, usability | Google Forms |
- – Mean sensitivity across nine shared drugs: GenTB-RF 77.6%, GenTB-WDNN 75.4%, TB-Profiler 74.4%, Mykrobe 71.9% 77.6% vs 75.4% vs 74.4% vs 71.9%
- – Specificity: Mykrobe 97.6%, TB-Profiler 96.9%, GenTB-WDNN 96.2%, GenTB-RF 96.1% 97.6% vs 96.9% vs 96.2% vs 96.1%
- ▼ Genotypic resistance sensitivity averaged across tools was lower for isoniazid and rifampicin on low-depth (<10x across 95% of genome) sequenced isolates 11% lower (isoniazid), 9% lower (rifampicin)
- – Average GenTB prediction pipeline runtime based on 23 runs 35 min (SD 4 min, median 35 min, IQR 33-38 min)
- – Pyrazinamide RF final model trained on variants occurring in phenotypically resistant isolates 393 variants in 3,262 isolates
- ▲ User survey rated GenTB favorably on clarity, pleasantness, stability, ease of use, and usability
- other 77.6% (95% CI 76.6-78.5%) (GenTB-RF mean sensitivity across nine shared drugs)
- other 75.4% (95% CI 74.5-76.4%) (GenTB-WDNN mean sensitivity across nine shared drugs)
- other 74.4% (95% CI 73.4-75.3%) (TB-Profiler mean sensitivity across nine shared drugs)
- other 71.9% (95% CI 70.9-72.9%) (Mykrobe mean sensitivity across nine shared drugs)
- other 97.6% (95% CI 97.5-97.7%) (Mykrobe specificity)
- other 96.9% (95% CI 96.7-97.0%) (TB-Profiler specificity)
- other 96.2% (95% CI 96.0-96.4%) (GenTB-WDNN specificity)
- count 20,408 isolates (Total validation dataset of sequenced isolates with phenotypic DST data)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper describes a diagnostic-accuracy benchmarking study rather than a classical hypothesis-testing experiment: two newly implemented machine-learning classifiers (GenTB-RF, GenTB-WDNN) and two existing tools (TB-Profiler, Mykrobe) were each used to predict drug resistance from 20,408 M. tuberculosis isolates with laboratory-based phenotypic DST as ground truth. Performance was summarized as sensitivity, specificity, and AUC per drug and tool, with 95% confidence intervals obtained by bootstrapping (resampling predictions). Runtime and a user questionnaire were reported descriptively (mean/SD, median/IQR, Likert scale), without traditional significance testing between tools or explicit multiplicity correction across the many drug/tool comparisons.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Bootstrap resampling to obtain confidence intervals | sensitivity, specificity, and AUC estimates for GenTB-RF, GenTB-WDNN, TB-Profiler, and Mykrobe, per drug | 5000 predictions per tool and drug on a resampled dataset | not stated |
| Out-of-bag (OOB) error minimization for hyperparameter tuning (mtry, classwt) | training/retraining of the pyrazinamide random forest model | 75%/25% train/validation split (15,267 / 5,098 isolates) | na |
| Descriptive summary statistics (mean, SD, median, IQR) | pipeline runtime evaluation | 23 completed pipeline runs | na |
| Descriptive Likert-scale summary | user-friendliness questionnaire (5 items: pleasing, clear, easy to use, stable, usable) | 12 respondents (of 166 surveyed) | na |
-
Diagnostic performance (sensitivity, specificity, AUC) of the four tools was compared using point estimates and bootstrapped 95% CIs, without a formal statistical test of whether differences between tools were significant.↳ Could also: A paired test for correlated proportions/ROC curves, such as McNemar's test for paired sensitivity/specificity differences or DeLong's test for comparing correlated AUCs — Since all four tools were run on the same set of isolates, the comparisons are naturally paired; a paired test would formally quantify whether observed differences in accuracy exceed what could arise from sampling variability alone.
-
Performance was assessed separately across 13 drugs and 4 tools without an explicit multiplicity adjustment for this large family of comparisons.↳ Could also: A false-discovery-rate procedure such as Benjamini-Hochberg, applied across the drug/tool comparisons — If formal significance tests were added to compare tools, an FDR correction would help control the overall rate of false positive findings across the many parallel comparisons.
-
95% confidence intervals for sensitivity and specificity (binomial proportions) were obtained via bootstrap resampling of predictions.↳ Could also: A closed-form binomial CI method such as the Wilson score interval or Clopper-Pearson exact interval — These methods provide an alternative, non-simulation-based way to compute confidence intervals for proportions and can be preferred for their direct analytic derivation, particularly for smaller subgroup sample sizes.
-
Pipeline runtime (based on 23 runs) was summarized with both mean/SD and median/IQR.↳ Could also: A bootstrap confidence interval around the median or mean runtime — For a modest sample size and a metric like runtime that can be right-skewed, a bootstrap CI would convey the uncertainty of the central estimate in addition to the reported dispersion measures.
-
The user questionnaire (Likert-scale responses, 12 respondents) was described narratively without a formal statistical summary or test.↳ Could also: Reporting median and IQR for each Likert item, and optionally a Wilcoxon signed-rank test against a neutral midpoint — Ordinal Likert data are often summarized with medians/IQR rather than means, and a signed-rank test offers a standard nonparametric way to assess whether responses differ from a neutral reference point.
-
Random forest hyperparameters (mtry, classwt) were selected by minimizing out-of-bag error and maximizing the sum of sensitivity and specificity on the same data used for model development.↳ Could also: Nested cross-validation with a held-out test fold separate from hyperparameter tuning — Nested cross-validation separates model selection from performance estimation, which can be used to reduce the risk of optimistic performance estimates when tuning and evaluation share data.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-34461978 (GenTB)
Paper: Gröschel et al. 2021, Genome Medicine. "GenTB: A user-friendly genome-based predictor for tuberculosis resistance powered by machine learning."
GenTB ships TWO ML predictors of M. tuberculosis drug resistance:
- GenTB-RF — per-drug random forests (R
randomForest), 992 SNPs/indels, trained on 1,397 isolates. - GenTB-WDNN — multitask wide-and-deep neural net (Keras 2.2.4 / TF 1.8 / Py3.6), trained on 3,601 isolates, 11 drugs (Chen et al. 2019 EBioMedicine).
Pipeline-derived results (IN SCOPE)
- R1 (WDNN, DETERMINISTIC): shipped trained model (
farhat-lab/wdnnmultitask_wdnn_021318.json+_weights_021318.h5) applied to the shipped independent validation set (validation_data.csv792 isolates × 222 features,validation_data_pheno.csv) → per-drug AUC. Reproduces the WDNN validation AUCs underlying GenTB-WDNN (GenTB Table 2 WDNN column / Chen 2019 validation). - R2 (WDNN, NON-DETERMINISTIC, harder):
evaluation.py5×10-fold CV retrain on the 3,601-isolate matrix (X_features.csv,labels.csv,alpha_matrix.csv) → cross-validated AUCs. Within-tolerance only (KFold shuffle unseeded + dropout). - R3 (RF, DETERMINISTIC pipeline):
gentb-sitebin/TBpredict.R+ shippedR/*_finalpredict.RData(all 13 drugs) on the shipped example genotype matrix → reproduce the shipped example prediction JSON (pipeline determinism check).
OUT OF SCOPE (infeasible / not shipped)
- Table 1/2 benchmark numbers on the 19,880-isolate external set: requires running the gentb-snakemake variant-calling pipeline on ~20k WGS raw-read accessions (Additional file 1) — petabyte-scale fetch + align, not feasible here. The benchmark genotype matrix is NOT deposited.
- Wet-lab DST phenotyping — external.
NOTE on metadata error
The brief's data accession zenodo:10.5281/zenodo.883859 is WRONG — that DOI
resolves to mwaskom/seaborn v0.8.1, unrelated. Real artifacts: code+models in the
GitHub repos (gentb-site, wdnn, gentb-snakemake); benchmark raw reads via NCBI/ENA
accessions in Additional file 1.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.