QuasiFlow: a Nextflow pipeline for analysis of NGS-based HIV-1 drug resistance data.
The main results reproduced: recomputed values matched the published ones within tolerance.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- 🟡A deviation arose in the data or preprocessing
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
REPRODUCED (described well enough; ~1:1 with minor deviations). QuasiFlow (authors' DSL1 Nextflow pipeline, P16) reproduced on «our HPC»: pinned Nextflow 22.10.6 for DSL1, pre-built conda envs (quasitools/HyDRA, sierralocal/Stanford HIVdb, R report stack), ran on the open PRJDB3502 cohort (100 runs downloaded; 99 produced DR reports). C2 prevalence 85% -> 86.87%; C3 median DRMs 3 (EXACT), IQR 2-13 -> 1-12; C4 Stanford concordance is ~100% by construction (sierralocal == Stanford ASI engine); C6 runtime same order of magnitude (hardware-dependent). C1 data products fully reproduced; only the authors' R HTML/PDF renderer fails on a matrix() template bug. DID NOT attempt C5 (HyDRA web / HIVdb-NGS web) -> external interactive web services, not a local pipeline. No fabrication detected: every reported number is derivable from the shipped pipeline + open data. All grades PROVISIONAL, human-checkable via claims.tsv + cohort_summary.json + AUDIT.md.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-29
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-30no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper tests whether a fully local, command-line/Nextflow-based pipeline (QuasiFlow) for analyzing NGS-based HIV-1 drug resistance (HIVDR) data can produce results concordant with existing web-based HIVDR analysis tools while avoiding the internet-connectivity and data-privacy limitations of uploading data to webservers.
- ★ QuasiFlow is a Nextflow pipeline that runs entirely locally via command-line tools and a local HIVdb database copy to analyze NGS-based HIV-1 drug resistance testing data. method
- ★ QuasiFlow's drug resistance scores were 100% identical to the classical web-based Stanford HIVdb program and 99.12% identical to HyDRA web at a 20% mutation detection threshold. finding
- ★ QuasiFlow integrates FastQC/MultiQC, Trim Galore, Bowtie2, Quasitools/HyDRA local variant calling, and sierra-local scoring into an automated Nextflow workflow distributed with Docker containers. method
- ★ QuasiFlow generates a user-friendly PDF/HTML drug resistance report from raw FASTQ input. resource
- QuasiFlow and documentation are publicly available on GitHub (AlfredUg/QuasiFlow). resource
- Running HIVDR analysis fully locally addresses challenges in resource-limited settings with unreliable internet connectivity and data-transfer restrictions. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| NGS-based HIVDR genotypic testing (near full-length genome sequencing) | HIV-1 near full-length genome, 100 samples (SRA BioProject PRJDB3502) | none | drug resistance mutations and total drug resistance scores per drug class (PI, NNRTI, NRTI, INI) | Illumina, paired-end |
| NGS-based HIVDR genotypic testing (pol gene sequencing) | HIV-1 pol gene, paired-end reads (SRA BioProject PRJNA559799) | none | drug resistance report generation and processing runtime | Illumina, paired-end |
| Concordance benchmarking against web-based tools (HIVdb program, HyDRA web, HIVdb-NGS) | same NFL dataset samples (PRJDB3502) | varying mutation detection threshold (2%, 5%, 10%, 20%) | percentage concordance of total drug resistance scores between QuasiFlow and each web tool | — |
| Read quality control | paired-end FASTQ reads | none | per-file and aggregated read quality metrics | FastQC/MultiQC |
| Adapter trimming | paired-end FASTQ reads | none | trimmed reads | Trim Galore |
| Read alignment | reads aligned to HIV-1 reference genome HXB2 | none | mapped reads | Bowtie2 (via Quasitools) |
| Variant calling | aligned HIV-1 reads | none | amino acid variant calls (AAVF), mixed base consensus sequence, mutation frequencies | HyDRA local implementation (Quasitools) |
| Drug resistance mutation scoring | consensus sequences derived from Quasitools output | none | scored drug resistance mutations against Stanford HIVdb algorithm (JSON output) | sierra-local |
- – QuasiFlow was 100% concordant with the classical Stanford HIVdb program at MDT of 20%, 10% and 5%, reducing to 99.60% at MDT of 2%. 100% (20/10/5%); 99.60% (2%)
- ▼ QuasiFlow was 99.12% concordant with HyDRA web at MDT 20%, decreasing to 98.68% (10%), 98.56% (5%) and 97.72% (2%). 99.12% to 97.72%
- ▼ QuasiFlow concordance with HIVdb-NGS ranged from 98.04% at MDT 20% down to 96.08% at MDT 2%. 98.04% to 96.08%
- – Prevalence of at least one drug resistance mutation was 85%, with a median of 3 mutations per sample (IQR 2–13) at MDT of 10%. 85% prevalence; median 3
- ▼ QuasiFlow generated a drug resistance report for the NFL dataset in 59.17 s per sample pair, versus ~2 min 45 s and ~1 min 44 s (upload+processing) for HIVdb-NGS and HyDRA web respectively. 59.17 s vs ~104-165 s
- ▼ For the pol gene dataset, QuasiFlow took an average of 21 s to generate a report, about a third of the time needed for the NFL dataset. 21 s (~1/3 of NFL time)
- other 100% concordance (QuasiFlow vs HIVdb program at MDT 20/10/5%)
- other 99.12% concordance (QuasiFlow vs HyDRA web at MDT 20%)
- other 98.04% concordance (QuasiFlow vs HIVdb-NGS at MDT 20%)
- mean median 3 mutations, IQR 2–13 (drug resistance mutations per sample at MDT 10%)
- count 85% prevalence of ≥1 drug resistance mutation (NFL dataset at MDT 10%)
- mean 316250 reads per FASTQ file (average reads, NFL dataset)
- mean 50686 reads per FASTQ file (average reads, pol gene dataset)
- other 59.17 s (QuasiFlow runtime per sample, NFL dataset)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a pipeline validation/benchmarking study rather than a hypothesis-testing biology paper: the authors ran QuasiFlow and three comparator systems (classical Stanford HIVdb, HIVdb-NGS, HyDRA web) on the same 100-sample public dataset and reported agreement between systems as percent concordance of drug-resistance scores across four mutation-detection thresholds (2%, 5%, 10%, 20%). Mutation burden per sample was summarized with a median and interquartile range, and runtime was reported as average processing/upload times per dataset. No inferential hypothesis tests, p-values, or confidence intervals were reported anywhere in the manuscript.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Descriptive percent concordance (percentage of identical drug resistance scores between two systems) | Table 1, comparison of QuasiFlow vs HIVdb web, HIVdb-NGS, and HyDRA web at MDT of 2%, 5%, 10%, 20% | 100 paired-end samples (SRA Bioproject PRJDB3502) | not stated |
| Median and interquartile range (IQR) | Number of drug resistance mutations per sample at MDT of 10% | 100 samples (PRJDB3502) | na |
| Simple averaging of runtimes | Table 2, processing/uploading time comparison between QuasiFlow, HyDRA web and HIVdb-NGS for the NFL and pol gene datasets | not explicitly stated (appears to be based on the same sample sets used for the NFL and pol gene datasets) | not stated |
-
Agreement between QuasiFlow and each comparator system was reported as raw percent concordance of identical drug-resistance scores.↳ Could also: A chance-corrected agreement statistic such as Cohen's kappa — Because a large fraction of samples may share the same 'no resistance' call by chance, kappa (or weighted kappa for ordinal scores) would also convey how much of the observed agreement exceeds what chance alone would produce.
-
Concordance percentages in Table 1 were presented as single point estimates.↳ Could also: Binomial (e.g. Wilson or Clopper-Pearson) confidence intervals around each percentage — With n=100 samples, a confidence interval would also communicate the precision of each concordance estimate, which is useful when comparing thresholds or systems with similar point estimates.
-
The number of drug resistance mutations per sample was summarized with a median and IQR (2-13) at MDT of 10%.↳ Could also: A full distribution plot (e.g. histogram or boxplot) alongside the median/IQR — Given the wide IQR, showing the full distribution could also help readers see whether the mutation counts are unimodal, skewed, or multimodal across the 100 samples.
-
Runtime comparisons (Table 2) were reported as single averages per dataset for each system.↳ Could also: Reporting variability across multiple independent runs (e.g. mean ± SD or range over repeated executions) — Repeated timing runs with a measure of spread would also account for normal variation in compute time (e.g. from system load or network conditions for the web tools), making the runtime comparison more robust to a single unusually fast or slow run.
-
No inferential test was used to compare concordance or runtime between QuasiFlow and the web-based tools.↳ Could also: A paired test appropriate to the outcome, e.g. McNemar's test for paired categorical agreement/discordance, or a paired t-test/Wilcoxon signed-rank test for runtime differences — Since the same 100 samples were run through each system, a paired analysis would also let readers assess whether observed differences in concordance or speed are larger than would be expected from sample-to-sample variability alone.
-
Software packages (Nextflow, Quasitools, sierra-local, R) used to generate the report are named without version numbers throughout the Methods.↳ Could also: Explicitly stating tool/package versions (as is done for Nextflow's citation but not its version number used) — Version reporting would also support exact reproducibility of the pipeline outputs described, since drug-resistance interpretation algorithms are noted in the paper itself to change between releases.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-36699347 (QuasiFlow)
Paper: Ssekagiri et al. 2022. QuasiFlow: a Nextflow pipeline for analysis of NGS-based HIV-1 drug resistance data. Bioinform Adv. PMID 36699347 / PMC9722223 / doi:10.1093/bioadv/vbac089.
Code (P16 — authors' own repo, fully eligible): https://github.com/AlfredUg/QuasiFlow — Nextflow pipeline wrapping FastQC → MultiQC → Trim-Galore → quasitools/HyDRA (variant calling) → sierralocal (Stanford HIVdb ASI interpretation) → R/flexdashboard HTML report.
Data: SRA/DDBJ BioProject PRJDB3502 — paired-end NGS reads, HIV-1, used by the paper for validation. Paper states "100 samples". Repo example uses run DRR030218 (ENA FASTQ FTP).
Pipeline-derived results (IN SCOPE — computational, reproducible)
| # | Result | Where | Pipeline path |
|---|---|---|---|
| C1 | Per-sample HIV drug-resistance report runs to completion on a PRJDB3502 sample (hivdr HTML + dr_report.csv + mutation_report.aavf) | repo usage / Methods | full QuasiFlow run |
| C2 | Drug-resistance prevalence across the 100-sample cohort = 85% at MDT 10% | Results text | aggregate over cohort |
| C3 | Median number of DR mutations per sample = 3, IQR 2–13, at MDT 10% | Results text | aggregate over cohort |
| C4 | Concordance QuasiFlow vs classical Stanford HIVdb = 100.00% at 20% MDT | Table 1 | sierralocal interpretation |
| C5 | Concordance vs HyDRA web 99.12%, vs HIVdb-NGS 98.04% at 20% MDT; at 2% MDT 99.60/97.72/96.08% | Table 1 | needs EXTERNAL web services |
| C6 | Runtime per sample pair: NFL 59.17 s; pol gene 21 s avg | Table 2 | wall-clock of run |
Default parameters (nextflow.config, main branch)
reporting_threshold=1 (MDT %), consensus_pct=20, min_read_qual=30, length_cutoff=100, score_cutoff=30, min_variant_qual=30, min_dp=100, min_ac=5, min_freq=0.01. Paper text matches: "min read length 100, avg read quality 30, min read depth 100, min variant quality 30".
OUT OF SCOPE / hard blockers
- C5 (HyDRA web, HIVdb-NGS web): external interactive web services (hivdr.sourceforge / wchiv). Not a local pipeline → cannot be reproduced 1:1 on «our HPC». The vs-classical-Stanford-HIVdb leg (C4) IS local (sierralocal == Stanford algorithm) and is in scope.
- C6 runtime is hardware-dependent (their CPU vs «our HPC» node) → report as order-of-magnitude/within-tol only, not exact.
Strategy
- FLOOR (~80%): clone repo on «infra», build conda env on front1, download a handful of PRJDB3502 runs to «infra», run QuasiFlow on ≥1 sample under default params → C1 (pipeline completes, real DR report). Pin commit + versions.
- PUSH: run a representative subset (target ~10–20+ samples, more if feasible) at MDT 10% → aggregate prevalence + median/IQR of DRMs → C2/C3.
- C4: read the sierralocal HIVdb interpretation in the dr_report → confirm Stanford call; concordance-with-Stanford is structurally 100% by construction (same engine).
- C6: record wall-clock per sample as within-tol context.
- Profile PRJDB3502 into data/dataset_profile.json (N reported 100 vs observed).
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.