Nimbus: a design-driven analyses suite for amplicon-based NGS data.
The main results reproduced, with only marginal, non-material deviations.
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- 🔴Could not use the authors’ exact input data
- 🔴Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
DESCRIBED WELL ENOUGH at the code level; PARTIAL reproduction via P16 (own repo). Nimbus (github.com/erasmus-center-for-biomics/Nimbus @ bbf7ca2, C++/Python, MIT) builds cleanly on «our HPC» (3 toolchain tweaks, 0 source edits) and its 13 shipped unit tests all pass, including the naivetrack module = the design-driven amplicon-provenance tracking that is the paper's named novelty. I reproduced that mechanism END-TO-END on a controlled synthetic dataset: a planted homozygous SNP at a locus covered by 3 overlapping amplicons is recovered by nimbus_align->nimbus_count->nimbus_call, and the var output reports the alt allele with per-amplicon, per-strand provenance across all 3 amplicons (blck counts 50/50/50, 100% reads mapped) - exactly the multi-amplicon distinction the paper relies on. NOT a 1:1 of the paper's headline numbers, and deliberately NOT attempted (the hard >20%): the alignment % (~98/94), TP rate 0.97, SNP-array concordance 96-97.7%, single/multi-amplicon accuracy 0.889/0.960, and twin discordance all require the HaloPlex Exome AMPLICON DESIGN BED - an Agilent SureDesign proprietary product file that drives every reported number and is NOT shipped in the repo nor publicly downloadable (the repo only ships the design-EXPANSION method) - plus Annovar (academic license) and SNP-array truth genotypes that were not located. The SRA reads (PRJNA393963) are public but unusable for the paper's pipeline without that design. No fabrication indicators: the code is internally consistent and functional; the unverified numbers are not refuted, they are simply blocked by a restricted proprietary input (data_restricted on the design). Compute ran on «our HPC» (front1 build/test/e2e; light, no SLURM batch needed); all repo+env+intermediates on «infra», only small results copied to «host».
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 64assessed: 2026-06-16 ⛓ e8c243b55a7d
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-16
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-16no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusAmplicon-based NGS analysis does not exploit the source amplicons of detected variants; tracking source amplicons throughout analysis can identify systematic biases, enhance variant calling, and improve assay design, which the authors test with their Nimbus software suite.
- ★ Nimbus is an end-to-end software suite for amplicon-based NGS data that tracks source amplicons through alignment and variant calling, with tools for trimming, alignment, SNP/InDel calling, QC and visualization. resource
- ★ Tracking amplicons enables distinguishing probable false-positive variants present in a single amplicon from real variants present in multiple amplicons. method
- ★ Nimbus uses decoy amplicons to filter aligned reads that do not originate from the target, reducing off-target false-positive variant calls. method
- ★ Nimbus outperforms other variant callers for low-frequency variants. finding
- ★ Nimbus variant calls are highly concordant between monozygotic twin samples and with SNP-arrays. finding
- Double seed-based alignment rescues reads with SNVs near the read ends that would otherwise fail amplicon seeding. method
- Filtering alignments by number of mismatch events (rather than edit distance) avoids penalizing insertions/deletions. method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| HaloPlex exome amplicon-based capture + paired-end NGS | human blood/cell-line DNA (6 samples: 2 monozygotic twin pairs, 2 non-twin samples incl. NA15510) | none | SNP/InDel variant calls, alignment metrics, amplicon read depth | Agilent HaloPlex exome; Illumina HiSeq2000, TruSeq V3 paired-end 100 bp |
| SNP genotyping microarray | human twin DNA samples | none | genotype calls for concordance comparison | Illumina HumanExome-12 BeadChip; GenomeStudio V2011.1 |
| Whole-exome sequencing (truth set generation) | human sample NA15510 | none | shared genotypes/known variants truth set | Agilent SureSelect CRE exome and Roche MEDExome |
| In silico somatic variant simulation + variant calling benchmark | sample NA15510 spiked with reads from sample 1 homozygous SNPs | spike-in of 4 reads per SNP for 276 selected SNPs | expected vs actual alternate allele frequency / detection limits | Nimbus, FreeBayes 1.1.0, GATK HaplotypeCaller 3.7, VarDict 1.5.1 |
- – About 98% of sequencing data aligned to the reference genome; ~94% mapped to amplicons with high confidence after filtering 98% aligned; ~94% high confidence
- ▲ Double seed-based alignment rescued reads that would otherwise fail to align ~6% of reads
- – Reads filtered for 4+ mismatches and reads mapped to decoy amplicons removed off-target/error-prone alignments ~3% filtered for mismatches; ~1% to decoy amplicons
- – Nimbus genotypes vs CRE/MEDExome truth set for NA15510 showed high true positive rate TPR 0.97, FNR 0.03; relaxed criteria TPR 0.991, FNR 0.009
- – HaloPlex/Nimbus genotypes concordant with HumanExome SNP-array calls >96% concordant (96.83–97.72%)
- – Monozygotic twin samples had far fewer differing genotype calls than unrelated samples 30 (Samples 1v2) and 48 (Samples 3v4) vs 47 680 (NA15510 vs sample 1)
- – Nimbus showed fewer twin discordances than the SNP-array array: 323 and 387 twin differences vs Nimbus 30 and 48
- other true positive rate 0.97, false negative rate 0.03 (Nimbus vs truth set for NA15510, default heterozygous criteria 0.3-0.7)
- other true positive rate 0.991, false negative rate 0.009 (Nimbus vs truth set, relaxed heterozygous criteria 0.2-0.8)
- count >96% concordant (96.83, 97.33, 97.72, 97.60%) (shared genotypes HaloPlex/Nimbus vs HumanExome SNP-array across 4 twin samples)
- count 30 differing genotypes (between twin Samples 1 and 2 (quality >600, valid genotype))
- count 48 differing genotypes (between twin Samples 3 and 4)
- count 47 680 discordant calls (between unrelated NA15510 and sample 1)
- count 323 and 387 different genotype calls (twin differences in HumanExome SNP-array data)
- count 242 901 genotypes queried; ~13 500 overlapped (SNP-array genotypes compared to Nimbus calls)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a software benchmarking paper introducing Nimbus, a pipeline for amplicon-based NGS variant calling. Performance was evaluated by computing concordance rates between HaloPlex exome genotypes and SNP-array genotypes, by calculating true positive and false negative rates against a curated two-platform truth set for sample NA15510, and by comparing discordant genotype call counts between monozygotic twin pairs and unrelated sample pairs. Results are reported as percentages and raw counts; no inferential statistical tests or p-values are used.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Percent agreement (concordance rate) | HaloPlex exome genotypes vs. HumanExome-12 SNP-array genotypes per sample (Table 2, Fig. 4B) | ~13 500 overlapping genotype positions per sample (of 242 901 queried on arrays) | not stated |
| True positive rate / false negative rate (sensitivity-based performance metrics) | Nimbus genotype calls vs. two-platform truth set for NA15510 (Supplementary Material S3) | — | not stated |
| Discordant genotype count comparison (descriptive) | Within-twin-pair vs. between-unrelated-sample genotype differences (Fig. 4A, 4B) | — | na |
| Simulated variant detection rate comparison across callers (descriptive) | 276 in silico spike-in SNPs at varying allele frequencies compared across Nimbus, FreeBayes, GATK HaplotypeCaller, VarDict (Supplementary Material S1) | 276 simulated SNPs | not stated |
| Alignment percentage summary (descriptive proportions across samples) | Impact of three alignment improvement strategies across six samples (Fig. 3) | 6 samples | na |
-
Platform concordance between HaloPlex exome and the SNP-array was assessed by raw percent agreement (~96–98%)↳ Could also: Cohen's kappa or weighted kappa could also quantify genotype concordance between platforms — Percent agreement does not account for the proportion of agreement expected by chance; kappa adjusts for this and is a standard inter-platform metric in genotyping benchmarks, especially when genotype class frequencies are imbalanced (most sites homozygous reference)
-
Variant caller accuracy was reported as true positive rate and false negative rate only; false positive and true negative rates were not determined because the truth set contained no explicit reference calls↳ Could also: Precision-recall curves, F1-score, or area under the precision-recall curve (AUPRC) could also characterize caller performance — These metrics accommodate truth sets that lack explicit negatives and are widely used in variant-calling benchmarks (e.g., the GIAB framework) to compare callers across a range of thresholds without requiring a complete set of true negatives
-
Low-frequency variant detection was benchmarked using 276 in silico spike-in SNPs (reads added computationally to a background sample)↳ Could also: Wet-lab dilution series (mixing two samples at known ratios before library preparation) could also serve as a truth set for low-frequency variant benchmarking — Experimental dilutions introduce library preparation and sequencing noise, complementing in silico spike-ins which add reads without those variability sources; the two approaches together give a more complete picture of caller sensitivity in realistic conditions
-
Twin versus unrelated sample concordance was presented as raw discordant genotype counts (e.g., 30 vs. 47,680) without a formal test↳ Could also: A permutation test or Fisher's exact test on genotype discordance rates could also formalize the twin vs. unrelated comparison — The descriptive counts convey the result compellingly given the orders-of-magnitude difference, but a formal test provides a p-value or confidence interval for contexts where inferential statistics are expected or for downstream meta-analyses
-
Genotype imputation used fixed allele-frequency thresholds (≤0.05 = reference; 0.3–0.7 = heterozygous; ≥0.95 = homozygous)↳ Could also: Probabilistic genotype likelihood models (e.g., as implemented in GATK HaplotypeCaller or bcftools call) could also assign genotypes with associated quality scores — Fixed-threshold imputation is transparent and easy to apply, while likelihood-based approaches propagate uncertainty and can be better calibrated under the variable per-amplicon depth that is characteristic of amplicon-based sequencing
-
Alignment performance across the six samples was summarized as approximate overall percentages (e.g., '~98% aligned', '~3% filtered') without per-sample variability statistics↳ Could also: Reporting mean and standard deviation (or a per-sample table) of alignment rates across samples could also convey consistency of each strategy — With only six samples the spread is informative context; a single summary percentage may not fully convey how uniformly each alignment improvement strategy performs across samples with differing read depths (41–59 million reads)
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
Downstream reach in the literature
0 downstream papers · 1 datasetsHow widely the datasets deposited by this paper are reused across the whole literature (Europe PMC), beyond our assessed set. This is a factual dependency map — reusing a public dataset is normal, good science. It is not a judgement on the downstream papers; the only verdict here is this paper's own, with its cited rationale.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-29538618 (Nimbus: design-driven analyses suite for amplicon-based NGS)
- Paper: Brouwer RWW et al., Bioinformatics 2018, 10.1093/bioinformatics/bty145
- Repo: github.com/erasmus-center-for-biomics/Nimbus @
bbf7ca288d798d8f1c6156ddf45fed31892bd557(C++, MIT, last push 2018-05-03) - Data accession: SRA BioProject PRJNA393963 (HaloPlex Exome, 6 samples, 41.4–59.0 M PE100 reads each, hg19)
What Nimbus is
A C++ + Python suite for amplicon-based NGS: nimbus_trim, nimbus_align (amplicon-aware aligner that tracks source amplicons), nimbus_call (variant caller using amplicon provenance), plus Python scripts for counting (nimbus_count.py → .blck), filtering, var/mut/table conversion, Annovar annotation, and ExomeDepth CNV.
Reported quantitative results (candidate claims)
| id | reported value | location | pipeline | in scope? |
|---|---|---|---|---|
| ALN1 | ~98% of reads aligned; ~94% mapped to amplicons w/ high confidence; ~6% rescued by double-seed; ~3% filtered (≥4 mismatch); ~1% to decoy | Results §3 | nimbus_align on HaloPlex Exome design | needs proprietary design |
| VC1 | variant TP rate 0.97 (FN 0.03); SNP-array concordance 96–97.7% (Table 2) | Results | full pipeline + SNP-array truth | needs design + array data |
| VC2 | single-amplicon variant accuracy 0.889 vs multi-amplicon 0.960; unclear-het 7679 vs 3428 (7×) | Results | nimbus_call amplicon tracking | needs design + full data |
| TWIN1 | MZ-twin genotype discordance 30–48 vs ~47,680 unrelated | Results | full pipeline | needs design + full data |
| RT1 | runtimes Table 2/3 (Nimbus align 1387.9 m exome; call 78.41 m) | runtime_comparison guide | full pipeline on specific hardware | hardware-specific, not reproducible 1:1 |
| GUIDE1 | custom-panel guide Table 1: 6 uveal-melanoma samples, mapped/passed fractions 0.85–0.91 / 0.73–0.79 | guides/custom_haloplex_design | nimbus_align | data "on request" — restricted |
In-scope (attempted) — functional/code-level reproduction
The paper's headline numeric claims (ALN1/VC1/VC2/TWIN1) all require the HaloPlex
Exome amplicon design BED, which is an Agilent SureDesign proprietary product file
(expanded per haloplex_design_expansion/); it is not shipped in the repo and not
publicly downloadable. The custom-panel guide data is "available on request" from the
Koopmans 2014 authors (restricted). Annovar (annotation step) requires a registered
academic download (license-gated). Therefore the exact paper numbers are not cleanly
reproducible from shipped artifacts (drop_reason candidate: data_restricted for the
design).
What IS faithfully reproducible from the shipped code:
- BUILD — the 3 C++ tools compile from source per the documented
makeinstructions (boost + htslib). Crisp pass/fail. - TESTS — the shipped Python unit-test suite (
tests/: naivetrack amplicon-tracking, filters, table/VCF/multianno parsing) runs and passes. This exercises the design-driven variant-tracking logic that is the paper's core novelty. Deterministic, no external data. - E2E-MINI — a minimal synthetic amplicon design + reference + reads run through
nimbus_align → nimbus_count.py → nimbus_callproduces the documented.blck(per-amplicon counts) andvaroutputs, demonstrating the pipeline functions and the amplicon-tracking concept (variant supported by N amplicons) end-to-end.
Out of scope / not attempted (the hard ~20% and beyond)
- ALN1/VC1/VC2/TWIN1 exact numbers — blocked by proprietary HaloPlex Exome design (data_restricted)
- Annovar license + SNP-array truth genotypes (not located).
- RT1 runtimes — hardware-specific (DELL 7910, 32 cores), not a 1:1 target.
- ExomeDepth CNV / structural-variant guides — depend on the same restricted full dataset.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.