Sequencing of human genomes with nanopore technology.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- Nothing in this column.
- 🔴Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough to reproduce the CODE ARTIFACT, not the whole-genome numbers. SEW (the authors' shipped phasing tool, bioconda r-sew 1.0.1, repo tag 1.0.1) installs and runs cleanly on «our HPC» and correctly phases its bundled example (chr10, 50 het SNPs) into haplotypes; an independent VCF-vs-truth switch-error check = 0% (confirmed with no-phasefile control, so not circular), and WhatsHap 2.8 on the same input also = 0% — reproducing the paper's headline switch-error METRIC and its SEW-vs-WhatsHap COMPARISON (qualitatively SEW ~= WhatsHap). NOT 1:1 on the paper's reported magnitudes: the example is 50 simulated SNPs, far easier than NA12878 chr22, so it cannot reproduce 1.84%/1.91% nor the sub-2% gap. NOT ATTEMPTED (out of 80/20 scope): 273.4 Gb yield, read N50/coverage (basecalling of raw ONT), SNV F1 93.4% (bwa mem + FreeBayes), exact chr22 switch error, clinical SAMD9L phasing (EGA controlled access) - because the raw->BAM/variant pipeline is 'available on request' (not public) and processing 273 Gb is the hard 20%. No fabrication flags; the gap is a reproducibility-surface limitation, not evidence of fabrication.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 75assessed: 2026-06-16 ⛓ 4eef7179109c
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-16
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-16no human curator yet
- Last updated
- 2026-09-19
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper evaluates whether Oxford Nanopore Technologies' MinION long-read sequencer can be used for routine whole-genome sequencing, including clinical applications, and whether a novel phasing method can improve its otherwise modest single-nucleotide variant calling accuracy.
- ★ A novel reference panel-free, read-based phasing algorithm substantially improves SNV calling accuracy over standard filtering in ONT data. method
- ★ Two non-synonymous de novo variants in SAMD9L were identified and directly phased to the same paternal haplotype in a clinical ataxia-pancytopenia sample. finding
- ★ Consensus SNV-calling error rates from ONT data remain substantially higher than those from short-read sequencing methods. finding
- ★ Phasing- and annotation-based filtering improves SNV F1 score, FDR, and FNR relative to quality/contamination filtering alone. finding
- ★ The novel phasing algorithm builds on the STITCH genotype imputation model, using per-haplotype per-SNP emission probabilities. mechanism
- ★ False-positive SNV calls are disproportionately enriched among putative loss-of-function and clinically relevant variants compared to true positives. finding
- Systematic biases (e.g., homopolymer-associated deletion errors) rather than random per-read error dominate residual SNV-calling errors, as shown by comparison to idealized simulations. finding
- Large structural variant calling from ONT reads using Sniffles identifies deletions with variable support across other sequencing technologies. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Whole-genome nanopore sequencing | NA12878 (GM12878 lymphoblastoid cell line) | none | read yield, read length, error rates, coverage depth | ONT MinION, R9.4 flow cells, Albacore v2.0.2 base-caller |
| SNV discovery/calling | NA12878 | none | F1 score, FDR, FNR versus GIAB truth set | FreeBayes |
| Read-based haplotype phasing | NA12878 | none | switch error rate versus parentally-phased truth data | novel phasing algorithm and WhatsHap |
| Structural/large variant discovery | NA12878 chromosome 22 | none | deletion calls versus multi-technology consensus truth set | Sniffles |
| Whole-genome nanopore sequencing and variant phasing | Patient with ataxia-pancytopenia syndrome and immune dysregulation (SAMD9L) | none (clinical de novo variants) | phase of two de novo protein-coding SAMD9L variants | ONT MinION |
| Base-caller comparison | NA12878 reads | different base-calling algorithms | per-read substitution/insertion/deletion error rates, downstream SNV accuracy | Albacore v2.0.2, Metrichor, and other base-callers |
| Simulated sequencing data analysis | Simulated NA12878 datasets | idealized random per-read error model, varying error rates and depth | F1 score, FDR, FNR across simulated error rates and coverage | — |
- – Mean per-read substitution rate of 12.7%, deletion rate of 4.7%, and insertion rate of 3.2% in aligned high-quality reads.
- – Average per-base coverage depth of 81.7x, with 90.4% of the genome covered by at least 40 reads. 81.7x
- ▲ Phasing- and annotation-based filtering improved whole-genome SNV F1 score from 88.3% (QUAL+contamination) to 93.4%, with FDR reduced from 10.9% to 5.3% and FNR from 12.5% to 7.8%. 5.1 percentage point F1 gain
- ▼ Novel phasing method achieved a lower switch error rate than WhatsHap (1.84% vs 1.91% overall on chr22; 0.80% vs 0.90% for spanned adjacent heterozygous SNPs). ~0.07-0.10 percentage point reduction
- – Two de novo non-synonymous SAMD9L variants were directly phased and found to lie on the same paternal haplotype.
- ▲ False positive variants were enriched among putative LoF/pathogenic sites (69/45219, 0.15%) compared to true positives (173/788782, 0.02%). ~7.5-fold higher proportion
- – Idealized simulations at matched depth/error rate achieved F1 of 99% (FDR/FNR ~1%), roughly 10 percentage points higher than observed real-data F1, indicating systematic (e.g., homopolymer) errors dominate residual inaccuracy. ~10 percentage point gap
- – Of 82 large variants (deletions) called on chr22 by Sniffles, 22 matched the truth set; of the remaining 60, 21 were PacBio-supported, 31 had ONT-only support, and 8 appeared to be false positives.
- mean substitution rate 12.7%, deletion rate 4.7%, insertion rate 3.2% (per-read error rates, NA12878 aligned high-quality reads)
- mean 81.7x average per-base coverage depth (NA12878 genome-wide coverage)
- other F1 88.3% (FDR 10.9%, FNR 12.5%) pre-phasing vs F1 93.4% (FDR 5.3%, FNR 7.8%) post-phasing, all autosomes (SNV calling accuracy before/after phasing-based filtering)
- other switch error rate: our method 1.84% vs WhatsHap 1.91% (all chr22 SNVs); 0.80% vs 0.90% (spanned adjacent hets) (phasing accuracy comparison, NA12878)
- count 45,740,123 total reads; 273.4 Gb total sequence yield across 73 flow cells (NA12878 sequencing yield)
- count 69/45219 (0.15%) false positives vs 173/788782 (0.02%) true positives among putative LoF variants (FDR impact on pathogenic variant enrichment)
- fold_change F1 99% with FDR/FNR 1% in idealized simulation vs ~93% F1 in observed data (simulated versus real ONT SNV-calling performance)
- count 82 large variants called on chr22, 22 matching truth set, 21 PacBio-supported, 31 ONT-specific, 8 apparent false positives (Sniffles large variant discovery on chromosome 22)
Statistical methods review
Model: opusA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a genomics technology-evaluation study that benchmarks Oxford Nanopore MinION whole-genome sequencing against established truth sets rather than testing biological hypotheses with classical inferential statistics. Performance is reported descriptively using classification accuracy metrics (F1 score, false discovery rate, false-negative rate, switch error rate, consensus accuracy) for SNV calling, phasing, and large-variant detection, with parameters tuned on chromosome 22 and then applied genome-wide. Comparisons between methods (base-callers, phasing algorithms, filtering approaches) are presented as point-estimate differences in these metrics, and enrichment of false positives among loss-of-function/constrained-gene variants is reported as raw counts and proportions.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Classification performance metrics (F1 score, FDR, FNR) vs GIAB truth set | SNV discovery in NA12878, chromosome 22 and full autosomes (Table 1, Fig. 2) | — | na |
| Consensus accuracy (percent) vs GIAB reference | Whole-genome SNV calling accuracy (99.9%) | — | na |
| Switch error rate vs trio-phased truth set | Phasing accuracy comparison of WhatsHap vs novel method, chromosome 22 | — | na |
| Raw count / proportion enrichment comparison | FP vs TP enrichment among putative LoF variants and constrained (pLI) genes (e.g. 69/45219 vs 173/788782) | counts of variant calls as stated | not stated |
| Simulation-based evaluation of performance across error rates and depths | Idealized-model NA12878 simulations (Supplementary Figs. 15–20) | — | stated (idealized random per-read substitution error, no amplification bias) |
| Visual/manual adjudication of calls against orthogonal platforms | Large-variant (>100 nt) calls on chromosome 22 vs PacBio/Illumina reads | 82 discovered variants | na |
-
Method and platform comparisons (e.g. WhatsHap switch error 1.91% vs the novel method 1.84%; base-caller comparisons) are reported as point-estimate differences in metrics.↳ Could also: Reporting these metrics with bootstrap or binomial confidence intervals, or a paired comparison across genomic intervals/flow cells. — Interval estimates would convey the uncertainty around each metric and help readers gauge how large or stable the observed differences are, which is especially informative when two values are close.
-
Variant-calling parameters were tuned to maximize F1 on chromosome 22 and then applied to the remaining autosomes.↳ Could also: A cross-validation or held-out chromosome scheme, or repeating the tuning across multiple folds. — An explicit train/test split or cross-validation would characterize how well tuned parameters generalize and quantify any optimism from selecting parameters on the same metric being reported.
-
Enrichment of false positives among LoF and constrained-gene variants is presented as raw counts and proportions (e.g. 17 FP vs 20 TP).↳ Could also: A Fisher's exact test or chi-square test with an odds ratio and confidence interval for the enrichment. — A formal association statistic plus effect size would put a number on the strength and uncertainty of the enrichment alongside the descriptive counts, particularly where counts are small.
-
Per-read and per-flow-cell error/coverage characteristics are summarized primarily as means (e.g. mean substitution rate 12.7%) and distributions in figures.↳ Could also: Accompanying central estimates with a dispersion measure such as SD, IQR, or range across flow cells. — An explicit spread measure would convey flow-cell-to-flow-cell variability directly in the text, complementing the distributional figures.
-
Large-variant calls were adjudicated by visual inspection against orthogonal platforms.↳ Could also: Computing sensitivity/precision against the consensus truth set with confidence intervals, or independent re-scoring by multiple blinded reviewers with an agreement statistic. — Quantified validation metrics and inter-reviewer agreement would make the manual adjudication step more reproducible and easier to compare across studies.
-
Differences between observed and simulated performance are interpreted by comparing point estimates (e.g. a 10-percentage-point F1 gap).↳ Could also: Running multiple simulation replicates and summarizing the distribution of outcomes. — Replicated simulations would provide a sense of Monte Carlo variability around the simulated metrics, clarifying how much of the observed gap exceeds simulation noise.
Result convergence & founder nodes
Findings this paper shares with others that ran a comparable experiment. A node’s strength is how many independent papers report it (replication breadth) — not how often it is cited, so a heavily-replicated but under-cited founder still stands out.
-
Read-based phasing of nanopore reads yields a low switch error rate on chr22 (1.84%), lower than WhatsHap.long-read-DNA human na12878 down 2019×1papers★ This paper is the founder (earliest)
-
Nanopore false-positive calls are enriched among putative pathogenic loss-of-function variants relative to true positives (0.15% vs 0.02%).long-read-DNA human na12878 up 2019×1papers★ This paper is the founder (earliest)
-
Nanopore whole-genome sequencing achieves 99.9% genome-wide consensus SNV accuracy on the full autosomes after pre-phasing quality/contamination filtering.long-read-DNA human na12878 2019×1papers★ This paper is the founder (earliest)
-
Haplotype phasing plus heuristic filtering improves full-autosome nanopore SNV calling (F1 to 93.4%).long-read-DNA human na12878 up 2019×1papers★ This paper is the founder (earliest)
-
Sniffles discovers 82 large structural variants (>100 nt) on chr22 from nanopore reads, most validated against truth/PacBio sets.long-read-DNA human na12878 2019×1papers★ This paper is the founder (earliest)
-
Nanopore sequencing identifies and phases two non-synonymous de novo SAMD9L variants to the same paternal haplotype in a patient with ataxia-pancytopenia syndrome.long-read-DNA human 2019×1papers★ This paper is the founder (earliest)
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-31015479
Paper: Bowden R, Davies RW, et al. Sequencing of human genomes with nanopore technology. Nat Commun 2019. PMID 31015479 · PMC6478738 · DOI 10.1038/s41467-019-09637-5.
Shipped code: https://github.com/Genomicsplc/SEW — SEW ("reference-panel-
free long-read phasing"), an R/C++ package by the same author (RW Davies),
derived from STITCH. On bioconda as r-sew v1.0.1. This is the only publicly
runnable code artifact from the paper.
Data: ENA PRJEB30620 (NA12878 ONT, 273.4 Gb raw); EGA EGAS00001003469 (clinical sample, controlled access). Code availability statement: SEW phasing code is public; "The code used to analyse the data ... are available on request from the authors."
In scope (attempted)
| Result | Pipeline | Feasible? |
|---|---|---|
| SEW long-read phasing tool runs & phases haplotypes | SEW (shipped) on its bundled example (chr10, 50 het SNPs, 1 sample, truth phasefile) | YES — small, deterministic, self-contained |
| Switch error rate metric (paper's headline phasing accuracy measure) | SEW output vs truth phasefile → switch error rate; WhatsHap on same input for the SEW-vs-WhatsHap comparison the paper reports | YES on the example data (mirrors the method, not the NA12878 numbers) |
Out of scope (NOT attempted, with reason)
| Result | Why dropped from attempt |
|---|---|
| Yield 273.4 Gb, read N50/mean 6,373 bp, coverage 81.7× | Derived from raw ONT signal across 73 flow cells (273 Gb). Basecalling = instrument/wet-lab-adjacent; not in repo. Out of 80/20. |
| Mapping (bwa mem 0.7.12), SNV F₁ 93.4% (FreeBayes v1.0.2) | Requires the full 273 Gb dataset + GIAB truth comparison; analysis code "available on request" (not public). Heavy 20%. |
| NA12878 chr22 switch error 1.84% / 0.80% (SEW) vs 1.91% / 0.90% (WhatsHap) | Exact numbers need chr22 ONT reads (basecalled, bwa-mapped) + het calls + GIAB truth. The raw→BAM pipeline is not shipped; processing 273 Gb is the hard 20%. We reproduce the metric & the SEW-vs-WhatsHap comparison on the tool's bundled example instead, and record the reported numbers for human audit. |
| Clinical SAMD9L phasing (199 kb block, 33 reads) | EGA controlled-access data + clinical specifics; out of scope. |
Approach
Run the authors' shipped SEW tool (the published, citable code artifact) on its own example data on «our HPC»; compute switch error rate vs the bundled truth, and run WhatsHap on the same input to reproduce the paper's qualitative claim that SEW's switch error is comparable to / slightly better than WhatsHap. This is a faithful 1:1 reproduction of the code artifact and its core metric, explicitly NOT a reproduction of the NA12878 whole-genome numbers (different, much larger data; basecalling + mapping code not public).
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
The shipped SEW tool reproduces cleanly: it installs, phases all 50 bundled het sites, and an independent VCF-vs-truth check gives 0/49 switch error (confirmed non-circular via a no-phasefile control), with WhatsHap 2.8 also at 0/49 — so the metric and the SEW≈WhatsHap comparison reproduce qualitatively. The gap is a reproducibility-surface limitation on the data/code side: the paper's chr22 numbers (1.84%/0.80%), 273.4 Gb yield, 81.7x coverage and 93.4% SNV F1 require a basecalling+mapping+variant pipeline that is only 'available on request' plus EGA-controlled clinical data, and the curator self-chose an easier toy example under 80/20 scope. Deviation is moderate and explainable (different/easier input, not different logic) with no fabrication signal — every reproduced value is regenerable from the public tool. Overall yellow: a solid code-artifact reproduction whose absolute magnitudes simply weren't checkable.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at [email protected].
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.