poreCov-An Easy to Use, Fast, and Robust Workflow for SARS-CoV-2 Genome Reconstruction via Nanopore Sequencing.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- Nothing in this column.
- 🟡Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
REPRODUCED 1:1. Tool/workflow paper reproduced via P16: ran the authors' own poreCov 1.1.0 (Nextflow 22.10.8 + apptainer, profile test_fastq,local,singularity) on its shipped OSF test data on «our HPC» («job», 16 cores, 10m02s, exit 0). All four in-scope pipeline claims reproduced exactly: C1 consensus is BYTE-FOR-BYTE identical to the author-shipped expected medaka consensus (29903 bp, 189 N, identity 1.00000, 0 diffs); C2 president QC PASS (identity 0.993>=0.90, N 0.63%<=5%); C3 pangolin lineage B.1.177; C4 nextclade clade 20E (EU1). Determinism comes from poreCov 1.1.0 pinning pangolin 3.1.17/2022-01-05 and nextclade 1.10.1 in its containers (--update NOT used). The dataset (OSF kf54a fastq + 87bc9 truth) is open, complete, and delivers what it promises as a CI/validation set. NOT attempted (out of scope): Table 1 (external GISAID metadata, not pipeline-derived), Table 2 (hardware-dependent runtimes), and Fig 2 per-sample/negative-control values (the 4 clinical samples are not publicly deposited; no ENA/SRA accession). All grades are provisional and human-checkable.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-18 ⛓ ca82027949f7
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-24
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper addresses whether a highly parallel, containerized Nextflow workflow (poreCov) can make accurate, reproducible SARS-CoV-2 genome reconstruction and lineage determination from nanopore (ARTIC protocol) sequencing data accessible to labs lacking dedicated bioinformatics expertise, while remaining fast and robust at high sample throughput.
- ★ poreCov is an easy-to-use, fast, and robust Nextflow-based workflow for reference-based SARS-CoV-2 genome reconstruction and lineage determination from nanopore sequencing data resource
- ★ poreCov wraps all required tools in version-controlled Docker/Singularity containers, requiring only Nextflow plus a container engine as dependencies method
- ★ poreCov flexibly accepts fast5, fastq, or fasta inputs and automatically adjusts workflow steps accordingly method
- ★ poreCov generates a comprehensive HTML/XLSX/TSV summary report combining QC, contamination, lineage, and coverage information for rapid interpretation resource
- ★ Sample-barcode bleeding can be identified by comparing SNP proportion patterns of negative controls against barcoded samples in BAM files finding
- Longer 1,200 bp amplicon primer sets reduce amplicon dropouts and increase throughput relative to standard 400 bp ARTIC V3 amplicons finding
- ★ poreCov has been validated at scale across multiple institutions, processing over 2,000 real SARS-CoV-2 clinical samples finding
- Requiring barcodes on both read ends mitigates but does not eliminate sample-barcode bleeding, at the cost of more unclassified reads finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| GISAID metadata survey | 1,185,291 publicly deposited SARS-CoV-2 genome entries | none | proportion of entries by sequencing technology used | GISAID |
| Workflow runtime benchmarking | poreCov pipeline (Nextflow) run on 10 fastq read files | varying hardware configuration (threads/RAM) and cores flag | time to final report | Docker/Singularity on local servers and Google Life Sciences cloud |
| Real-world clinical sample sequencing and genome reconstruction | SARS-CoV-2 PCR-positive patient samples from Jena University Hospital, University Hospital Regensburg, and Germany's Public Health Institute | none | number of samples/genomes successfully processed | MinION/GridION nanopore sequencers |
| Taxonomic read classification | nanopore sequencing reads from patient samples | none | proportion of human, viral (SARS-CoV-2), and unclassified reads | Kraken2 against combined GRCH38.p13 + GISAID SARS-CoV-2 database |
| Read quality control | fastq (basecalled) or fast5 (raw) nanopore reads | none | read length distribution, quality scores | NanoPlot (fastq) / pycoQC (fast5) |
| Reference-based genome reconstruction and variant calling | length-filtered amplicon nanopore reads mapped to Wuhan reference (NC_045512.2) | none | consensus genome sequence, masked low-coverage regions | minimap2, SAMtools/BCFtools, medaka, Longshot, nanopolish (ARTIC pipeline) |
| Lineage and clade assignment | reconstructed/polished SARS-CoV-2 consensus genomes | none | lineage, clade, mutations, deletions, genome quality metrics | pangolin, nextstrain, president |
| Sample-barcode bleeding assessment | negative controls vs. barcoded samples (BAM alignment files) | overloaded DNA molecule input during library prep | SNP base-proportion patterns per position | IGV / Unipro UGENE visualization of BAM files |
- – Illumina was the dominant sequencing technology for SARS-CoV-2 surveillance, followed by nanopore Illumina 77.61% (n=913,228) vs Nanopore 17.21% (n=202,503) of 1,185,291 GISAID entries
- ▼ poreCov runtime decreased substantially with more parallel hardware resources 1 h 23 min 52 s (4 threads) to 7 min 47 s (256 threads), ~10.8-fold reduction
- – poreCov was used in production across multiple institutions on large real-world sample sets >600 samples (Jena), >400 samples (Regensburg), 1,012 samples (Germany's Public Health Institute, Feb 23–May 5, 2021)
- – A negative control showed reads classified as SARS-CoV-2, indicating possible contamination or barcode bleeding 98 reads
- – Genomes are flagged QC pass based on identity and ambiguous base thresholds ≥90% identity to Wuhan reference and ≤5% ambiguous nucleotides (Ns)
- – Low-coverage genome regions are masked during reconstruction coverage ≤20x masked
- ▼ Typical samples yield high proportions of viral reads, while lower proportions correlate with reconstruction failure or low quality typical ≥90% viral reads
- count 1,185,291 GISAID entries screened (April 2021) (sequencing technology survey)
- other Illumina 913,228 (77.61%); Nanopore 202,503 (17.21%); SMRT 30,447 (2.59%); Ion Torrent 12,895 (1.10%); DNBSEQ 12,658 (1.08%); Sanger 4,898 (0.42%) (proportion of sequencing technologies used for SARS-CoV-2 surveillance)
- other runtime from 1 h 23 min 52 s to 7 min 47 s depending on hardware (4 to 256 threads) (poreCov time-to-results benchmarking, 10 fastq files)
- count 98 reads classified as SARS-CoV-2 in a negative control (example report showing possible contamination/barcode bleeding)
- count >600 samples analyzed at Jena University Hospital (real-world validation of poreCov)
- count >400 samples analyzed at University Hospital Regensburg (real-world validation of poreCov)
- count 1,012 samples analyzed at Germany's Public Health Institute between Feb 23 and May 5, 2021 (real-world validation of poreCov)
- other QC pass threshold: ≥90% identity to NC_045512.2 and ≤5% ambiguous nucleotides (genome quality control criteria used in poreCov report)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a software/methods paper presenting poreCov, a Nextflow-based workflow for SARS-CoV-2 genome reconstruction from nanopore sequencing data; no formal inferential statistical tests are reported. Evaluation is descriptive: runtime benchmarks across five hardware configurations are presented as single-measurement durations (Table 2), and sequencing technology proportions from GISAID are reported as counts and percentages (Table 1). Pipeline validation is conveyed through deployment scale (>600 samples at Jena University Hospital, >400 at University Hospital Regensburg, and 1,012 samples at the Robert Koch Institute) without comparative statistical analysis.
-
Runtime benchmarks in Table 2 are reported as single wall-clock measurements per hardware configuration↳ Could also: Repeated benchmark runs (e.g., n ≥ 3 per configuration) summarized with median and interquartile range or mean ± SD could also characterize throughput — A single timing measurement captures one realization of a stochastic process influenced by I/O scheduling, background load, and container caching; repeated measurements would quantify run-to-run variability and allow comparison intervals rather than point estimates
-
Pipeline performance is validated by reporting the number of samples processed at three sites without a formal comparison to existing tools (e.g., nf-core/viralrecon, ncov2019-artic-nf)↳ Could also: A head-to-head benchmarking study on a shared reference dataset with known variant calls could also characterize relative accuracy, sensitivity, and specificity — Deployment counts demonstrate feasibility and scale but do not quantify concordance with a ground truth; a comparative benchmark on, for example, simulated reads or WHO reference panel samples would allow readers to situate poreCov's accuracy relative to other workflows
-
QC pass thresholds are fixed at ≥90% nucleotide identity to the Wuhan reference and ≤5% ambiguous bases (Ns) without derivation↳ Could also: A sensitivity analysis varying these thresholds across the >1,000 processed samples, or a receiver-operating characteristic approach against an orthogonal quality metric, could also characterize how threshold choice affects the pass/fail rate — Reporting the empirical distribution of percent identity and N-content across the full sample set would allow readers to understand how conservative or permissive the chosen thresholds are relative to the data actually observed
-
Proportions of sequencing technologies in Table 1 are reported as raw counts and percentages from a GISAID snapshot (April 2021)↳ Could also: Confidence intervals for each proportion (e.g., Wilson score intervals) could also be reported alongside the point estimates — With n > 1 million entries, the proportions are estimated with high precision, but explicitly reporting uncertainty would be standard practice for descriptive epidemiological proportions and makes the precision transparent
-
The negative control contamination example (98 reads classified as SARS-CoV-2) is presented qualitatively via visual inspection of SNP proportion patterns in IGV↳ Could also: A formal contamination threshold (e.g., reads-above-background defined by a Poisson or binomial model) could also be derived from multiple negative control runs — A quantitative threshold would give users a reproducible criterion for flagging contamination rather than relying on visual inspection, and would allow the sensitivity of the negative-control check to be characterized across different sequencing runs
-
Sample sizes at the three deployment sites are reported as lower-bound estimates (>600, >400, 1,012) without breakdown by variant, primer version, or sequencing device↳ Could also: Stratified reporting by ARTIC primer version (V3 vs. 1,200 bp), device type (MinION vs. GridION), or time period could also characterize pipeline performance across subgroups — Pooled deployment counts demonstrate broad applicability but do not reveal whether performance differed across protocol variations; stratified summaries would be informative for labs choosing between primer sets or devices
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-34394197 (poreCov)
Paper: Brandt et al. 2021, poreCov—An Easy to Use, Fast, and Robust Workflow for SARS-CoV-2 Genome Reconstruction via Nanopore Sequencing, Front Genet, DOI 10.3389/fgene.2021.711437.
What poreCov is: a Nextflow workflow (DSL2) that wraps existing tools to go from Nanopore reads → SARS-CoV-2 consensus genome → QC + lineage/clade. Wrapped tools (poreCov 1.1.0 pinned containers):
- ARTIC pipeline
artic minion --medaka(genome reconstruction) —nanozoo/artic:1.3.0-dev - minimap2 (mapping, inside ARTIC), medaka (consensus polishing), Longshot/medaka variants
- president
rkibioinf/president:v0.6.3(identity-to-reference + N% QC) - pangolin
nanozoo/pangolin:3.1.17--2022-01-05(lineage) - nextclade
nanozoo/nextclade:1.10.1--2022-01-18(clade + mutations) - kraken2 (read decontamination, OPTIONAL, OFF by default), NanoPlot/pycoQC, Krona, CoVarPlot
This is a tool/workflow paper → P16 reproduction: run the authors' own tool on its own shipped test data and verify the described pipeline-derived behaviour.
Reported results and their reproduction status
| # | Reported result | Location | In scope? | Pipeline |
|---|---|---|---|---|
| Table 1 | GISAID sequencing-tech distribution (Illumina 77.61% n=913,228; nanopore 17.21% n=202,503 of 1,185,291) | Table 1 | OUT — external GISAID metadata snapshot, not pipeline-derived | — |
| Table 2 | Runtime benchmark (4thr/32GB 1h23m; 24thr/64GB 23m; 256thr/1TB 7m47s; GCloud 19m) | Table 2 | OUT (1:1) — hardware-dependent; we report our wall-clock as a non-comparable sanity datum only | nextflow |
| Fig 2 | Sample report for 4 clinical samples + 1 neg control (neg ctrl: 98 reads classified SARS-CoV-2 by kraken2); V1200 primers | Fig 2 | OUT — clinical samples NOT publicly deposited (no ENA/SRA); screenshot, no per-sample numbers in text | poreCov full |
| QC criteria | "QC pass" = ≥90% identity to Wuhan (NC_045512.2) AND ≤5% Ns; coverage masked at ≤20× | Methods | IN | president |
| Workflow behaviour | poreCov reconstructs a QC-passing SARS-CoV-2 consensus + assigns lineage/clade from Nanopore reads | Throughout | IN (core claim) | full poreCov |
In-scope reproduction (what we actually run)
Run poreCov 1.1.0 (the version in the documented test command) end-to-end on its
own shipped test data via the test_fastq profile:
- Test reads:
https://osf.io/kf54a/download→SARSCoV2.fastq.gz(183,965 Nanopore reads, sample2020_12_10_ukj_covidbarcode13, V1200/Midnight 1200 bp amplicons). - Built-in ground-truth consensus shipped by the authors:
https://osf.io/87bc9/download→SARSCoV2.fasta, header>barcode13/ARTIC/medaka MN908947.3(~29.9 kb) — this is poreCov's OWN medaka consensus for the same sample, so the fastq→consensus step has an author-provided expected output to compare against.
Reproduction claims:
- C1 (consensus reconstruction): reads →
artic minion --medakaconsensus; compare to shippedSARSCoV2.fasta(author ground truth). Metric: length + base identity. - C2 (QC / president): consensus identity to Wuhan NC_045512.2 ≥ 90% and Ns ≤ 5% → expect QC pass (the criterion the paper defines).
- C3 (lineage / pangolin 3.1.17, data 2022-01-05): lineage call for the consensus (deterministic at pinned container data version).
- C4 (clade / nextclade 1.10.1): clade + mutation count.
Determinism note: pangolin/nextclade calls drift with database version; poreCov 1.1.0
pins these in its containers (pangolin data 2022-01-05, nextclade 1.10.1), so the
calls are reproducible at the paper-era pin. We use -r 1.1.0 containers, NOT --update.
Out of scope (not attempted) and why
- Table 1 GISAID stats: external metadata, not produced by the pipeline.
- Table 2 timings: hardware-specific; no faithful 1:1 possible on different hardware.
- Fig 2 per-sample/neg-control numbers: the clinical samples are not publicly deposited (Data Availability points only to the GitHub repo; no
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
This is a tool/workflow paper (poreCov) reproduced legitimately by re-running the authors' own pinned pipeline (pangolin 3.1.17, nextclade 1.10.1) on its shipped OSF test data, which makes the capability claims deterministic and derivable. The gaps are on the data-availability/our-scope side, not the authors' side: the paper's clinical samples behind Table 1 and Fig 2 were never deposited, so those exact values are uncheckable and were placed out of scope. Severity is unresolved rather than large — the «our HPC» run («job») was still in progress with all claims PENDING and agreement.json not-run-yet. Net: a solid, explainable, but unfinished reproduction with no fabrication signal → yellow across the board.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.