VGEA: an RNA viral assembly toolkit.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No relevant deviation in data/preprocessing
- ✓No authors-side cause for any deviation
- ✓Any deviation was negligible
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
VGEA RNA-viral-assembly Snakemake pipeline reproduced on «our HPC». C0 integration test FULLY reproduced (exit 0; 2 consensus genomes + QUAST 99.59% + multiqc) after 4 documented env-rot fixes. C1 Table-2 read counts reproduced EXACTLY from decompressed FASTQs (=ENA spots x2; 1.8/2.0/3.2/4.4/6.2 M), cross-checked vs ENA metadata. C2/C3 (Table-2 contigs 28-42 + genome fraction 73-99.8%, = QUAST on IVA de-novo contigs): CV18 reproduced EXACTLY (42 contigs, 99.776% genome fraction, 0 misassemblies); CV29 99.860%/28c, CV115 89.188%/20c, CV145 87.543%/46c -- ALL genome fractions inside the paper's 73-99.8% band; contig counts differ in detail (IVA de-novo is non-deterministic; bioconda gives iva 1.0.8 vs paper tag 1.0.11). KEY FINDING: the prior rooms' 'IVA never converges on the 3 large samples' was actually a BROKEN iva conda env (pyfastaq import / strict-priority setuptools solve failure), NOT IVA runtime -- once the iva env is built with flexible channel priority (py3.9, iva 1.0.8), the samples assemble normally. With that fixed, CV18 + CV145 assembled cleanly; CV45 (the largest, 6.2M reads) assembled most of the way (iter29/seed26) but did not finish within the 11h IVA budget and was killed at the 12h cluster MaxWall cap (no contigs) -- IVA has no resume, so it cannot beat the cap by splitting. Net: 4/5 Table-2 samples reproduced (CV18 EXACT) + integration test + read counts. NOT a 1:1 bit-identical match (de-novo stochasticity) but a faithful reproduction. Out of scope (not attempted): Table-1 wall-clock/RAM benchmarks (hardware-dependent); shiver consensus + SPAdes comparison columns; Lassa/HIV-1 sets.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-18 ⛓ cd46167664a3
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-30
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetExisting standalone bioinformatics tools for RNA viral genome assembly can be combined into a single flexible, low-resource, easy-to-use Snakemake pipeline (VGEA) that performs comparably or better than established assembly tools while being deployable on a personal computer without HPC/cloud access.
- ★ VGEA is a Snakemake workflow that chains existing tools (fastp, BWA, SAMtools, IVA, shiver, SeqKit, QUAST, MultiQC) into an all-in-one RNA viral genome assembly pipeline resource
- ★ VGEA can be deployed on a personal computer without requiring HPC clusters or cloud-based systems, addressing access limitations in LMICs method
- ★ VGEA was successfully tested and validated on SARS-CoV-2 (n=20), HIV-1 (n=20) and Lassa virus (n=20) datasets finding
- ★ VGEA-generated contigs had the highest N50 across all five SARS-CoV-2 comparison datasets versus SPAdes and Velvet finding
- ★ VGEA used the least maximum RAM compared to SPAdes and Velvet across all five comparison datasets finding
- ★ SPAdes and Velvet completed assembly faster than VGEA across all analyses finding
- Mapping to shiver's sample-tailored constructed reference is systematically superior to mapping to the closest of 3,249 references finding
- VGEA requires no prior programming knowledge and can run automatically from input to final report with a single command method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| NGS-based reference-guided/de novo genome assembly pipeline (VGEA) | SARS-CoV-2 clinical samples | none | consensus genome assembly, contig/N50/genome fraction quality metrics | Illumina MiSeq |
| NGS-based genome assembly pipeline (VGEA) | Lassa virus clinical samples | none | consensus genome assembly quality | Illumina MiSeq / MiSeq FGx |
| NGS-based genome assembly pipeline (VGEA) | HIV-1 samples (NCBI SRA) | none | consensus genome assembly quality | Illumina HiSeq 2500 |
| Computational benchmarking (wall-clock time, CPU/RAM usage per pipeline rule) | SARS-CoV-2 sample dataset | none | time and maximum RAM per workflow rule | Snakemake benchmarking feature |
| Comparative genome assembly evaluation (VGEA vs SPAdes vs Velvet) | SARS-CoV-2 datasets (CV18, CV29, CV45, CV115, CV145) | assembly pipeline/tool | N50, NG50, genome fraction, misassemblies, mismatches, indels, max RAM | QUAST; SPAdes v3.15.2; Velvet v1.2.10 |
- ▲ VGEA contigs had the highest N50 scores across all five SARS-CoV-2 comparison datasets
- ▲ VGEA contigs had the highest genome fraction in all five datasets, covering >95% in four of them >95% in 4/5 datasets
- ▼ VGEA used the least maximum RAM of the three pipelines for all five comparison datasets
- – SPAdes and Velvet ran faster than VGEA for all analyses
- ▲ Mapping to shiver's constructed reference recovered missing sequence compared to mapping to the nearest of 3,249 references median 205 bases recovered
- ▲ Mapping to shiver's constructed reference produced more accurate base calls than mapping to the nearest reference median 13 bases called differently and more accurately
- – IVA de novo assembly step was the most time-consuming rule in the VGEA benchmark run 8:19:11 (h:m:s)
- – Human reference indexing step used the most RAM among VGEA pipeline rules in the benchmark run 4688.56 MB
- count 60 total validation datasets: SARS-CoV-2 (20), HIV-1 (20), Lassa virus (20) (VGEA validation dataset composition)
- other median 13 bases called differently and more accurately (shiver vs nearest-of-3249-references mapping comparison (cited from Wymant et al., 2018))
- other median 205 bases of missing sequence recovered (shiver vs nearest-of-3249-references mapping comparison (cited from Wymant et al., 2018))
- other genome fraction 99.776% (CV18), 99.786% (CV29), 98.291% (CV45), 96.957% (CV115), 73.093% (CV145) for VGEA (Table 2 assembly comparison, per-sample VGEA genome fraction)
- other IVA rule: time 8:19:11, max RAM 238.57 MB (Table 1 benchmarking of VGEA rules on SARS-CoV-2 sample)
- other human_reference_index rule: time 1:01:53, max RAM 4688.56 MB (Table 1 benchmarking of VGEA rules on SARS-CoV-2 sample)
- count reads per dataset (x10^6): CV18=3.2, CV29=1.8, CV45=6.2, CV115=2, CV145=4.4 (Table 2 input read counts for assembly comparison datasets)
- other CV18 VGEA: 42 contigs, N50=2294, NG50=2294, mismatches=10, indels=0, max RAM=627 MB (Table 2 per-pipeline assembly metrics for sample CV18)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a bioinformatics software paper presenting VGEA, a Snakemake pipeline for RNA viral genome assembly. Validation was performed on 60 publicly available sequencing datasets (n=20 each for SARS-CoV-2, HIV-1, and Lassa Virus). Pipeline performance was compared against SPAdes and Velvet on 5 SARS-CoV-2 datasets using QUAST-derived assembly metrics (N50, NG50, genome fraction, misassemblies, RAM, wall-clock time); results were reported as raw tabulated values with narrative summary, with no formal inferential statistics applied.
-
Pipeline comparison across the 5 SARS-CoV-2 datasets was made by visually inspecting tabulated QUAST metrics without formal statistical testing↳ Could also: A non-parametric paired test (e.g., Wilcoxon signed-rank or Friedman test with post-hoc comparisons) could be applied across the 5 datasets for each metric (N50, genome fraction, etc.) — Formal tests would allow quantification of whether observed metric differences between pipelines exceed what might be expected by chance across samples, and would be particularly informative if the comparison were extended to a larger dataset panel
-
The head-to-head comparison used only 5 SARS-CoV-2 datasets, while the full validation used 60 datasets (20 per virus)↳ Could also: The comparison could have been conducted on all 60 validation samples or stratified by virus type (SARS-CoV-2, HIV-1, LASV) — A larger and more diverse comparison set would provide stronger evidence of generalizability across different viral genome sizes, mutation rates, and sequencing depths
-
Runtime and RAM benchmarking were reported for a single representative SARS-CoV-2 sample with a single run↳ Could also: Benchmarking could be repeated across multiple samples and multiple runs per sample to report mean ± SD or range for each pipeline step — Single-run timing values do not capture run-to-run variability due to system load, I/O caching, or dataset-specific differences in read depth; summary statistics across replicates or samples would give a more robust resource estimate
-
Assembly quality metrics (N50, NG50, genome fraction) were compared on an absolute basis between pipelines without normalizing for input read depth↳ Could also: Metrics could be reported alongside or normalized by sequencing depth (reads per sample varied from 1.8M to 6.2M across the 5 comparison datasets) — Read depth is a known confounder of assembly contiguity; reporting metrics relative to depth (or stratifying by depth tier) helps distinguish pipeline effects from input-data effects
-
Performance was evaluated using QUAST metrics referenced against the SARS-CoV-2 reference genome for all three pipelines↳ Could also: Additional assembly-quality tools such as BUSCO (for gene completeness) or assembly graph inspection could complement QUAST's reference-based metrics — Reference-based metrics like genome fraction depend on the quality and completeness of the chosen reference; orthogonal reference-free or gene-content metrics can provide a complementary view of assembly completeness
-
The paper reported no measure of within-pipeline variability (e.g., across multiple assemblies of the same sample with different random seeds or parameter settings)↳ Could also: Sensitivity analyses varying key parameters (e.g., IVA thread count, shiver reference panel) could be reported for at least one representative sample — Documenting parameter sensitivity helps users understand how robust the reported performance metrics are to pipeline configuration choices, which is relevant given that VGEA is designed to be user-customizable
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
scope.md — pmid-34567846 (VGEA: an RNA viral assembly toolkit)
Paper
Oluniyi et al. 2021, PeerJ 9:e12129. VGEA = a Snakemake pipeline that assembles RNA-virus genomes from paired-end Illumina reads: fastp (trim/QC) → bwa (map to human) → samtools (extract non-human reads) → IVA (de novo assembly) → shiver (contig→ref align + read remap → consensus) → seqkit (tidy) → QUAST (assembly QC) → MultiQC (aggregate).
Code (P16: authors' own repo)
https://github.com/pauloluniyi/VGEA · pinned commit c9f81def52ab37eeb139600db6fd235700f5a841 (2021-09-12).
Ships: full Snakemake workflow, per-rule conda env pins, an integration test
(.tests/integration), and resources/paper_analyses/ with the EXACT sample
tables + configs used for the paper figures/tables.
Data
- SRA BioProject PRJNA666685 — 20 SARS-CoV-2 metagenomic Illumina runs (the paper's SARS-CoV-2 set). Author internal names CVxx map to SRR via BioSample alias.
- (also PRJNA666664 Lassa; HIV-1 from public SRA per paper). figshare 13009997 = data mirror.
- CV→SRR (Table-2 samples): CV18=SRR12749477, CV29=SRR12749476, CV45=SRR12749464, CV115=SRR12749458, CV145=SRR12749475.
IN SCOPE (pipeline-derived, attempted)
- Integration test (shipped, deterministic). Run
.tests/integration/test_config.yamlend-to-end with--use-conda; expect 2 consensus genomes + multiqc report. This is the CI's own test → quick MINIMUM that the pipeline-as-described runs. - Table 2 (real paper data). Run VGEA on the 5 SARS-CoV-2 Table-2 samples
(CV18/29/45/115/145) and compare QUAST metrics to reported ranges:
contigs 28–42, genome fraction 73–99.8%, highest N50; plus per-sample read
counts (paper: 1.8–6.2 M reads). - Read-count anchor. fastp/fastq read counts per sample vs SRA spot counts ×2.
OUT OF SCOPE (not attempted)
- Wet-lab sequencing (in-house MiSeq/FGx library prep, RNA extraction).
- Table 1 wall-clock/RAM benchmarks — hardware-dependent («our HPC» ≠ authors' machine), not 1:1 reproducible; reported as context only, not graded.
- SPAdes comparison columns in Table 2 require running a second assembler; attempt only if time permits (secondary).
- Lassa (PRJNA666664) and HIV-1 sets — same pipeline; attempt only after SARS-CoV-2.
Pipelines per result
All results above derive from the single VGEA Snakemake pipeline (tools pinned in workflow/envs/*.yaml). QUAST provides the comparison metrics.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
This is an in-progress reproduction of the authors' own VGEA Snakemake toolkit, not a finished one. Input data (PRJNA666685, 20/20 samples) is fully available and the only checked endpoint so far — Table 2 read counts — brackets the paper's 1.8-6.2M range (CV29=1.81M, CV45=6.24M). However, the contig counts (28-42), genome fraction (73-99.8%) and the shipped integration test are still PENDING («job»), so derivability and the core assembly claim are only partially confirmed; no discrepancy is observed and nothing points to the authors' side. Graded yellow overall purely because completion is pending, not because of any substantive deviation.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.