EDGE COVID-19: a web platform to generate submission-ready genomes from SARS-CoV-2 sequencing efforts.
Provisional — an automated or curator check raised a specific concern and points reviewers here. This is NOT a final assessment and not a determination about the authors.
The main result did not reproduce in this reproduction attempt. Where our recomputation produced values that differ from the published ones, those discrepancies are listed below. This is a single automated attempt — not peer review and not a finding of error or misconduct — and differences can also arise from data access, undocumented parameters or the computing environment. The verdict can be contested via “report an error”.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Reported values were directly comparable
- 🟡Could not use the authors’ exact input data
- 🟡A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
EC-19 is an applications-note web pipeline (FaQCs -> bwa/minimap2 vs NC_045512.2 -> variant call -> consensus -> Pangolin --usher). Its quantitative Results come NOT from the paper's own deposit PRJNA714680 (44 Illumina amplicon runs, 'dozens of genomes', no numeric claim) but from 9 OTHER reprocessed public bioprojects whose EC-19 output the authors deposited as LANL-data.txt. Tier-1 audit: re-derived the paper's aggregate numbers directly from that deposited table. STRONG reproductions: per-sample SNV rate 13.45 vs 13; linear coverage 86.5% vs 88%; min coverage 1.41% vs 1.4%. MISMATCHES flagged for human audit: deposited table has 386 samples vs 239 reported (superset); indel counts 2337 (6.05/sample) vs 139 (0.6/sample) ~10x higher because the deposit holds unfiltered ONT homopolymer insertions while the paper reports a filtered set whose filtering step is undocumented. Total SNVs (5192) scale to ~3115 once restricted to 239. NOT attempted: Table 1 EC-19-vs-Galaxy concordance (needs external Galaxy workflow). PENDING («our HPC» down centrally during this pass): Tier-2 = run EC-19 on individual accessions to validate single rows of LANL-data.txt, and on the 55 CDC benchmark SRRs to reproduce 100% lineage accuracy. Honest verdict: core per-sample claims reproduce; the deposited supplement does not 1:1 match the printed N and indel totals -> partial.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 49assessed: 2026-06-19 ⛓ 1f018ead43d9
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-25
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetCan an integrated, web-based bioinformatic platform (EDGE COVID-19) generate consistent, high-quality, submission-ready SARS-CoV-2 consensus genomes from both Illumina and Oxford Nanopore sequencing data, with variant and lineage calls that are accurate and concordant with established workflows like the COVID-19 Galaxy Project?
- ★ EDGE COVID-19 (EC-19) is a web-based platform that automates QC, reference-based variant/consensus calling, lineage determination, and submission of SARS-CoV-2 genomes and metadata to GenBank, GISAID and INSDC for both Illumina and ONT data. resource
- ★ EC-19 correctly assigned lineages for all 55 CDC benchmark datasets tested. finding
- ★ EC-19 shows strong concordance with the Galaxy workflow on Illumina data, with 95% agreement for both SNVs and indels. finding
- ★ EC-19 and Galaxy show greater discordance on ONT data, especially for indels (55% agreement) versus SNVs (95% agreement). finding
- ★ Most discrepancies between EC-19 and Galaxy occur in low-coverage samples/regions or near amplicon primer binding sites. mechanism
- EC-19 uses FaQCs for QC, Minimap2 (ONT) or BWA mem (Illumina) for mapping, and Pangolin with usher mode for lineage assignment. method
- EC-19 was used to successfully process and submit real-world New Mexico surveillance genomes to GISAID and GenBank. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Amplicon/shotgun sequencing (Illumina) with EC-19 workflow (QC, mapping, variant/consensus calling) | SARS-CoV-2 clinical samples, New Mexico, USA | none | consensus genome, SNVs/indels, submission success to GISAID/GenBank | Illumina |
| Amplicon sequencing (ONT) with EC-19 workflow | SARS-CoV-2 clinical samples, New Mexico, USA | none | consensus genome, SNVs/indels, submission success to GISAID/GenBank | Oxford Nanopore Technologies (ONT) |
| Lineage assignment benchmarking (Pangolin --usher mode) | 55 SRR benchmark datasets with known lineages (CDC) | none | concordance of predicted vs known lineage | Pangolin |
| Cross-workflow SNV/indel comparison (EC-19 vs Galaxy) | 88 Illumina SRA runs, ARTIC protocol, 6 Bioprojects/98 Biosamples | none | shared vs platform-specific SNV/indel calls, allele frequency, depth of coverage | BWA mem / Galaxy pipeline |
| Cross-workflow SNV/indel comparison (EC-19 vs Galaxy) | 151 ONT SRA runs, ARTIC protocol, 6 Bioprojects/98 Biosamples | none | shared vs platform-specific SNV/indel calls, allele frequency, depth of coverage | Minimap2 / Galaxy pipeline |
- – EC-19 correctly predicted all lineages in the CDC benchmark set 55/55 (100%)
- – Illumina SNV and indel calls agreed between EC-19 and Galaxy 95% agreement
- – ONT SNV calls agreed between EC-19 and Galaxy 95% agreement
- – ONT indel calls agreed between EC-19 and Galaxy 55% agreement
- – ONT deletion calls agreed between EC-19 and Galaxy 82% (69/84) agreement
- – Majority of Illumina workflow differences occurred in low-coverage regions (DP<100x) 59/68 differences
- – Test dataset quality across 239 samples avg 1834x fold coverage, 88% linear coverage
- – Total variants reported by EC-19 across all projects 3115 SNVs, 139 indels (avg 13 SNVs, 0.6 indels/sample)
- count 239 SRA datasets (151 ONT, 88 Illumina) (cross-workflow comparison dataset size)
- count 55/55 lineages correctly predicted (CDC benchmark lineage validation)
- other 95% agreement (Illumina SNV/indel concordance between EC-19 and Galaxy)
- other 95% agreement (SNV), 55% agreement (indel) (ONT SNV vs indel concordance between EC-19 and Galaxy)
- other 82% agreement (69 shared, 4 Galaxy-specific, 10 EC-19-specific) (ONT deletion concordance)
- mean average fold coverage 1834x; average linear coverage 88% (overall test dataset quality)
- count 3115 SNVs and 139 indels reported; avg 13 SNVs and 0.6 indels per sample (total EC-19 variant calls across all projects)
- count 36 low-quality samples with <75% linear genome coverage (subset of low-quality samples in comparison dataset)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a bioinformatics platform description and validation paper rather than a hypothesis-testing study. The authors validate the EC-19 workflow using benchmark and real-world sequencing datasets and compare its SNV, indel, and lineage calls against those from the COVID-19 Galaxy Project workflow. Results are reported as raw counts, percentages of agreement (concordance), and average values (e.g., mean depth of coverage, mean allele frequency) across samples, without inferential statistical hypothesis tests, p-values, or formal measures of uncertainty.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Descriptive concordance/percent agreement calculation (no inferential statistical test) | Comparison of SNV and indel calls between EC-19 and Galaxy workflows (Table 1, Section 4.4-4.5) | 239 SRA datasets (151 ONT, 88 Illumina) from six Bioprojects and 98 Biosamples | na |
| Simple accuracy tally (correct/total) | Lineage assignment validation against CDC benchmark datasets (Section 4.2) | 55 SRR datasets with known lineages | na |
-
Agreement between EC-19 and Galaxy workflows on SNV/indel calls is summarized as raw counts and percent agreement (e.g., 95% for Illumina SNVs).↳ Could also: A formal paired-agreement statistic such as Cohen's kappa (for categorical call/no-call agreement) or McNemar's test (for paired discordant pairs) could also be used — These measures quantify agreement while accounting for chance concordance and can accompany the descriptive percentages already reported, giving a standardized effect size for inter-workflow agreement.
-
Sequencing depth of coverage and allele frequency are summarized using single average values (e.g., average fold coverage of 1834x, average AF of 0.46 for certain SNVs).↳ Could also: Reporting the SD, IQR, or a 95% confidence interval alongside these means would also be a standard approach — A dispersion measure conveys how variable coverage and AF are across the sample set, which is relevant given the wide range noted (e.g., linear coverage from 1.4% to high values).
-
Discrepancies between workflows are attributed narratively to factors such as low depth of coverage or proximity to primer-binding sites.↳ Could also: A regression model (e.g., logistic regression predicting discordant call probability from depth of coverage, allele frequency, and primer distance) could also be used — This would let the relationship between these factors and call discordance be quantified with effect estimates and confidence intervals rather than described qualitatively.
-
Lineage assignment accuracy is reported as 55/55 correct calls with no accompanying interval.↳ Could also: A binomial confidence interval (e.g., Wilson or Clopper-Pearson) around the accuracy estimate could also be reported — With a relatively small benchmark set, an interval estimate communicates the plausible range of the true accuracy more fully than the point estimate alone.
-
The comparison dataset (239 SRA runs) was drawn from an existing collaborative resource (ACTIV TRACE) rather than a pre-specified statistical sample size.↳ Could also: For this type of workflow-validation/benchmarking study, using an opportunistic or convenience benchmark set is standard practice and formal power analysis is not typically expected — Noted here for completeness: unlike hypothesis-driven experiments, tool-validation studies commonly rely on available reference datasets rather than a priori sample-size calculations.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-35561186 (EDGE COVID-19 / EC-19)
Paper: Lo CC et al., EDGE COVID-19: a web platform to generate submission-ready genomes from SARS-CoV-2 sequencing efforts. Bioinformatics 2022. PMID 35561186 · PMC9113274 · DOI 10.1093/bioinformatics/btac176.
Type: Applications Note describing a bioinformatics web platform (EC-19). The
pipeline: FaQCs read QC → align to modified RefSeq NC_045512.2 (33-nt poly-A
trimmed) with minimap2 (ONT) / BWA-mem (Illumina) → variant calling
(indel AF>0.5 Illumina, >0.6 ONT, >0.8 in homopolymers; ambiguous base if ≤5
supporting reads) → consensus genome → Pangolin (--usher) lineage →
GenBank/GISAID/SRA submission packaging. JBrowse for visual inspection.
Code
- Main platform: https://github.com/LANL-Bioinformatics/EDGE (EC-19 is its SARS-CoV-2 workflow).
- Deposited results data: https://github.com/LANL-Bioinformatics/Lo-et-al-Bioinformatics-EC-19-data
→
LANL-data.txt= the per-variant EC-19 output for the public comparison datasets (the ground-truth supplementary table behind the Results numbers).
Data
- PRJNA714680 = the authors' OWN new data (New Mexico surveillance). ENA
read_run: 44 runs, all Illumina MiSeq AMPLICON. Paper calls them "dozens of genomes" — no specific reproducible numeric claim is attached to this accession. - The paper's quantitative Results (239 samples, SNV/indel/coverage totals,
Table 1 concordance) come from 9 OTHER public bioprojects that EC-19
reprocessed (PRJEB41737, PRJEB37886, PRJNA650037, PRJNA707211, PRJNA627229,
PRJNA613958, PRJNA614995, PRJNA628662, PRJNA616147), whose EC-19 output the
authors deposited in
LANL-data.txt. PRJNA714680 is NOT in that table.
In scope (pipeline-derived) — what we reproduce
- Tier 1 (done, no-compute): Re-derive the paper's aggregate Results numbers
(N, avg/total SNVs & indels, avg/linear coverage, low-quality count, min
coverage) directly from the authors' deposited
LANL-data.txtand compare to the printed values. This is a fabrication-detection audit: does the deposited data support the paper's numbers? → claims C1–C9. - Tier 2 (pending «our HPC»): Run EC-19 (or its documented equivalent:
minimap2/bwa→consensus→Pangolin) on a few accessions that have ground-truth
rows in
LANL-data.txt, plus the 55 CDC benchmark SRRs, to confirm per-sample variant calls / coverage / lineage reproduce the deposited values → C10, and individual-row validation of C2/C3/C6/C7.
Out of scope
- C11/C12 (Table 1 EC-19-vs-Galaxy concordance): requires also running the external Galaxy SARS-CoV-2 workflow and intersecting variant sets — not a single-pipeline reproduction; not attempted.
- Web-platform UI, JBrowse, automated GenBank/GISAID/SRA submission (manual / interactive / external services) — wet-lab/manual, out of scope.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
Tier-1 audit re-derived the paper's Results 4.3 aggregates from the authors' own deposited per-variant table. Reproduces cleanly: per-sample SNV rate (13.45 vs 13), linear coverage (86.5% vs 88%), min coverage (1.41% vs 1.4%) — good evidence these numbers are genuine. Does not match 1:1: the deposit is a 386-sample superset of the reported 239, and its indel totals are ~10x the printed figures (2337 vs 139) because the deposit is unfiltered and the filtering/subsetting steps are undocumented. The discrepancies sit on the authors' side (incomplete documentation, not derivable as printed) but are explainable and do not overturn the tool's central claim; the flagship 55/55 CDC lineage benchmark remains pending. Overall partial / yellow.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.