Evaluation of the Available Variant Calling Tools for Oxford Nanopore Sequencing in Breast Cancer.
Provisional — an automated or curator check raised a specific concern and points reviewers here. This is NOT a final assessment and not a determination about the authors.
The main results reproduced, with only marginal, non-material deviations.
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
PARTIAL methodology reproduction (P16) of a benchmark/evaluation paper of ONT variant callers on BRCA1/2. KEY DATA-AVAILABILITY FINDING: the deposited accession PRJNA865100 (run SRR20747220, 'BRCA-Sample') is ONE clinical patient amplicon that appears in NONE of the reported results; the headline benchmark (Tables 2-6) was computed on UNPINNED ont-open-data HG001/HG002 reads that were never deposited -> the paper's exact numbers are NOT independently verifiable from any shipped data (unverifiable-by-design, flagged; not fabrication). We re-ran the described pipeline on citable public ONT data (ont-open-data giab_2023.05 sup_60x HG001+HG002, BRCA1/2 subset) -> minimap2 2.22 map-ont -> samtools markdup -> {Clair3, Longshot, NanoCaller, Medaka} -> filter PASS+QUAL>20 -> hap.py(vcfeval) vs GIAB v4.2.1. RESULT: the high-SNP-accuracy REGIME reproduces - Clair3/Longshot/NanoCaller reach F1 mostly 95-100% with ~100% precision on BRCA1/2, the same regime the paper reports (90-98%); 8/28 claims land within tolerance (esp. HG002 + BRCA2: e.g. HG001 Clair3 BRCA2 SNP F1 97.98% vs 96.65%, HG002 Clair3 BRCA1 100% vs 97.17%). But exact per-tool values, variant counts and the caller RANKING diverge: in our newer R10.4.1 sup data Longshot ties/beats Clair3 (paper ranked Longshot 4th), and Medaka 2.x (haploid) technically failed (0 TP, NOT a true 0%). NOT attempted/ gradable 1:1: exact paper values (inputs undeposited), Clair v2.11, separate wf-human-SNP. Grade distribution: within-tol 8, partial 6, mismatch 10, error 3, not-attempted 1. Verdict: described-well-enough to REBUILD the method, but DIFFERENT from the paper on exact numbers because the paper's own inputs are not reconstructable; honest partial.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-29
- Rubric version
- not recorded
- Assessed by
- —
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetA systematic evaluation of six variant calling tools (Human-SNP-wf, Clair3, Clair, NanoCaller, Longshot, Medaka) for Oxford Nanopore long-read sequencing can identify which tool(s) most accurately detect SNPs and indels (validated against GIAB truth sets) to guide a targeted BRCA1/BRCA2 variant-detection workflow for breast cancer.
- ★ Clair3 and Human-SNP-wf (which incorporates Clair3) achieved the highest performance among the six variant callers tested. finding
- ★ An in-house targeted long-read sequencing pipeline for BRCA1/BRCA2 variant detection was developed and validated using GIAB reference samples. method
- Longshot uses a pair-Hidden Markov Model on long-read data to call SNPs but cannot detect indels. mechanism
- Medaka predicts SNPs from unphased long reads, then uses WhatsHap to phase reads before making final SNP/indel calls. mechanism
- ★ Clair3 combines pileup calling (fast, handles most candidates) with full-alignment calling (precise, for complex candidates). mechanism
- NanoCaller uses a deep convolutional neural network incorporating long-range haplotype structure to improve variant detection. mechanism
- ★ Hap.py with the vcfeval engine was used to benchmark caller outputs against GIAB v4.2.1 truth sets, producing TP, FP, FN, Precision, Recall, and F1-score. method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Long-read (Nanopore MinION) targeted sequencing and variant calling | GIAB reference samples HG001 (NA12878) and HG002 (NA24385) | none (reference/benchmark samples) | Precision, Recall, F1-score of called SNPs/indels vs GIAB truth set | Oxford Nanopore MinION |
| Read alignment | HG001/HG002 reads aligned to GRCh38/hg38 | none | SAM/BAM alignment output | Minimap2 |
| BAM processing (sort/index/PCR duplicate removal) | HG001/HG002 BAM files | none | sorted, indexed, deduplicated BAM files | Samtools (rmdup) |
| Coverage depth calculation | BRCA1 and BRCA2 target regions in HG001/HG002 | none | mean sequencing depth (X coverage) before/after PCR duplicate removal | Bedtools coverage |
| Variant calling comparison across six tools (Medaka, wf-human-snp, Clair3, Clair, Longshot, NanoCaller) | BRCA1/BRCA2 target regions, HG001 and HG002 | none (default parameters used for uniformity) | VCF output filtered by PASS and QUAL>20; TP/FP/FN, Precision, Recall, F1-score, runtime | respective tool pipelines; benchmarked with Hap.py (vcfeval engine) |
| Variant annotation and clinical significance lookup | Filtered SNV/INDEL VCFs from BRCA1/BRCA2 | none | predicted gene/amino-acid effects and clinical significance classification (wild type/VUS/PV/LPV) | SnpEff; ClinVar database |
- ▲ Clair3 and Human-SNP-wf achieved the highest Precision, Recall, and F1-score among the six tested variant callers.
- ▼ NanoCaller had the lowest (fastest) runtime for variant calling on BRCA1/BRCA2 regions in both HG001 and HG002.
- ▲ Clair had the longest runtime for variant calling in both HG001 and HG002 samples.
- – Minimum sequencing depth across samples never fell below 50X. ≥50X
- – For HG001, mean coverage before PCR-duplicate removal was 32.62X (BRCA1) and 36.89X (BRCA2); after removal, coverage for BRCA1 and BRCA2 was the same. 32.62X / 36.89X
- – For HG002, mean coverage before PCR-duplicate removal was 53.85X (BRCA1) and 70.06X (BRCA2); after removal, coverage for BRCA1 and BRCA2 was the same, suggesting the public reference data had already undergone deduplication or was whole-genome sequencing. 53.85X / 70.06X
- mean 32.62X (HG001 BRCA1 mean coverage before PCR-duplicate removal)
- mean 36.89X (HG001 BRCA2 mean coverage before PCR-duplicate removal)
- mean 53.85X (HG002 BRCA1 mean coverage before PCR-duplicate removal)
- mean 70.06X (HG002 BRCA2 mean coverage before PCR-duplicate removal)
- other QUAL > 20 (Quality threshold used to filter PASS variants for comparison across tools)
- other ≥50X (Minimum sequencing depth threshold observed across samples)
- count 6 (Number of variant calling tools evaluated (Human-SNP-wf, Clair3, Clair, NanoCaller, Longshot, Medaka))
- count 2 (Number of GIAB reference genome datasets used (NA12878/HG001, NA24385/HG002))
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This paper is a benchmarking/methods-comparison study rather than a hypothesis-testing study: six Nanopore variant-calling tools (Medaka, wf-human-snp, Clair3, Clair, Longshot, NanoCaller) were each run on two public reference genome datasets (HG001/NA12878 and HG002/NA24385) restricted to the BRCA1/BRCA2 regions, and their output VCFs were compared against GIAB high-confidence truth sets using the hap.py benchmarking tool. Results were reported as point-estimate performance metrics (true positives, false positives, false negatives, Precision, Recall, F1-score) and runtime, summarized per tool per sample in tables, with no inferential statistical hypothesis tests, p-values, or measures of uncertainty reported.
-
Tool performance was summarized as single point-estimate values of Precision, Recall, and F1-score per sample without any measure of uncertainty.↳ Could also: Bootstrap resampling of variant calls (or binomial/Wilson confidence intervals on Precision and Recall) could also be used — This would convey how much the observed Precision/Recall/F1 values might vary due to sampling of variant sites, giving readers a sense of the precision of the performance estimates alongside the point values.
-
The six variant callers were ranked and compared using raw metric values without a formal statistical comparison between tools.↳ Could also: A paired comparison approach, such as McNemar's test on concordant/discordant calls between tool pairs at the same genomic positions, could also be applied — This would allow a formal test of whether observed differences in calling accuracy between specific tool pairs are larger than would be expected by chance, complementing the descriptive metric comparison.
-
Evaluation relied on two reference samples (HG001 and HG002), each processed once per tool.↳ Could also: Including additional GIAB reference samples (e.g., HG003–HG007) or repeated resequencing runs could also be used — This would let performance rankings be assessed for consistency across a larger and more diverse set of samples, which is a common approach in variant-caller benchmarking studies.
-
Variants were filtered using a single fixed threshold (QUAL > 20, PASS-only) before computing Precision/Recall/F1.↳ Could also: A precision-recall or ROC-style curve across a range of quality thresholds could also be generated — This would show how the Precision/Recall/F1 trade-off shifts continuously with the filtering threshold, rather than reporting metrics at one chosen cutoff.
-
Mean sequencing coverage for BRCA1/BRCA2 was reported as a single average value per sample/gene, before and after PCR-duplicate removal.↳ Could also: Reporting the full distribution of per-base coverage (e.g., with SD or interquartile range, or a coverage plot) could also be used — This would communicate how uniform or variable coverage was across the targeted regions, beyond a single mean figure.
-
Runtime differences between tools (e.g., NanoCaller fastest, Clair slowest) were reported as single measured values per sample.↳ Could also: Repeated timing runs summarized with mean and SD (or a formal paired comparison of runtimes) could also be used — This would account for run-to-run variability in computational timing, which can be affected by system load, and give a more robust basis for comparing tool speed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-36140751
Paper: Helal AA, Saad BT, Saad MT, Mosaad GS, Aboshanab KM. Evaluation of the Available Variant Calling Tools for Oxford Nanopore Sequencing in Breast Cancer. Genes (Basel) 2022. PMID 36140751 · PMCID PMC9498802 · DOI 10.3390/genes13091583.
Nature of the paper: a benchmark / evaluation study. No authors' own analysis repository exists; the listed "code" is the third-party aligner minimap2 (github.com/lh3/minimap2). Per brief rule P16, applying the described third-party tools to the paper's data is an equally valid reproduction.
The pipeline (as described in Methods, Table 1)
- Basecalling: Guppy v5.0.16 (MinKNOW)
- Alignment: minimap2 v2.22 (ONT long-read preset) → GRCh38/UCSC hg38
- Post-processing: samtools v1.14 (sort/index,
rmdup -sPCR-dup removal), bedtools (coverage), custom BED targeting BRCA1 + BRCA2 - Variant calling (6 tools, default params "for uniformity"): Medaka v1.4.4 · Clair v2.11 · Longshot v0.4.1 · NanoCaller v2.1.2 · Clair3 v0.1-r11 · epi2me-labs/wf-human-SNP v0.3.1 (wraps Clair3)
- Filter: keep PASS, QUAL > 20
- Benchmark: hap.py v0.3.15 (vcfeval engine) vs GIAB v4.2.1 high-confidence truth for HG001/HG002, restricted to BRCA1/BRCA2
- Annotation/clinical: SnpEff v5.1 + ClinVar (annotation only)
Reported results (candidate claims)
- Table 2 — coverage per gene per reference sample (before/after dup removal)
- Table 3 / Table 4 — variant counts per caller per gene (HG001 / HG002)
- Table 5 / Table 6 — recall / precision / F1 per caller, split SNP vs INDEL, per gene, per reference sample (HG001 / HG002) + per-tool runtime
In scope (pipeline-derived, attempted)
- Recall / precision / F1 per caller (Tables 5/6) — the headline result.
- Variant counts per caller (Tables 3/4).
- Coverage (Table 2). All are computed by a bioinformatic pipeline → in scope.
Out of scope
- Wet-lab steps (DNA extraction, PCR amplification, MinION run, Guppy basecalling on raw FAST5) — not attempted; we start from public FASTQ/BAM.
- SnpEff/ClinVar clinical annotation narrative — descriptive, no benchmarked number.
CRITICAL DATA-AVAILABILITY FINDING (drives the verdict)
The paper's headline numbers (Tables 2–6) are computed on NA12878 (HG001) and NA24385 (HG002) ONT reads taken from the ONT open-data AWS registry (registry.opendata.aws/ont-open-data) — these are not pinned (no run IDs, file names, dates, or coverage given) and were not deposited.
The only data deposited by the authors — PRJNA865100 / SRR20747220 / SRS14394223 — is a single clinical breast-cancer patient amplicon sample ("BRCA-Sample" / library HITS-BRCA-1: human whole blood, 32 y female, ONT MinION 1D PCR amplicon of BRCA1/2). This patient sample appears nowhere in the paper's reported results (no patient table/figure; the clinical application is stated as "future plan"). It has no truth set, so no recall/precision can be computed from it.
Consequence for reproduction:
- The deposited accession does not contain the inputs behind any reported number → a strict 1:1 reproduction from "the paper's data" is not possible.
- We therefore (a) profile the deposited sample honestly, and (b) attempt a methodology reproduction of the benchmark (Tables 5/6) using publicly obtainable HG002/HG001 ONT data for BRCA1/BRCA2 vs GIAB v4.2.1, run through the same toolset. Because the exact source reads/coverage cannot be matched, this is graded partial at best (concordance of magnitude/ranking, not exact values).
- We additionally run the deposited patient sample through minimap2 + a caller to confirm the pipeline executes on the paper's own data (profiling/QC only — no reported number to grade against).
Regions (GRCh38)
- BRCA1: chr17:43,044,295–43,125,483
- BRCA2: chr13:32,315,086–32,400,268
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.