Ultra-deep multi-oncopanel sequencing of benchmarking samples with a wide range of variant allele frequencies.
Provisional — an automated or curator check raised a specific concern and points reviewers here. This is NOT a final assessment and not a determination about the authors.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓Reported values are derivable from the shared data
- ✓The central claim held under reproduction
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡The deviation was non-trivial in magnitude
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
SEQC2 multi-oncopanel benchmarking data descriptor (Gong et al. 2022, Sci Data). Described well enough to reproduce its headline computational result from the public deposit PRJNA677997, and this run added the FASTQ-level confirmation the previous pass lacked. (1) Metadata reproduction of Table 1 'Avg. read count' per panel from ENA/SRA metadata: 3/8 exact (IDT, IGT, TFS), 4/8 within ~5% (AGL, BRP, QGN, ILM>100M) after the spots-vs-reads convention, only ROC off (1.15x, post-UMI/subset). Run counts exact for the 4 main samples (400: A=103,B=103,C=91,AC5=103); 'extra Sample AIS' 104 vs 103 (+1). seqtk v1.0 IDT equalization: all 60 IDT runs = exactly 62,500,000 pairs (stdev 0). (2) NEW — FASTQ-level compute on «our HPC» (SLURM «job», node n125, seqtk 1.4 built from source, downloaded via «infra» proxy): 2 IDT runs each counted EXACTLY 62,500,000 read pairs (125M reads), confirming the seqtk equalization at the read level; single-end TFS run counted 8,048,625 = ENA read_count exactly, proving the 1x convention; ROC run = 57,978,005 per mate, confirming ENA read_count is the true raw FASTQ count (so the ROC table gap lies in what the paper counted, not the deposit). seqtk-count == ENA read_count for every file checked. NOT attempted (honest, out of quick scope): ~5000X/10000X read-depth (needs per-panel target-BED alignment) and VAF/variant-detection-limit validation (figshare truth VCFs + somatic calling). Dataset profiled same pass: grade A, complete, delivers as promised. Verdicts provisional; a human reviewer signs off.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 80assessed: 2026-06-19 ⛓ b2be82efd062
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-25
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper describes the development of four reference genomic samples with well-characterized variant content, used to enable a transparent, cross-laboratory, multi-oncopanel sequencing study assessing the sensitivity, false positive rate, and reproducibility of clinical oncopanels across a wide range of variant allele frequencies.
- ★ Four reference samples (Sample A, Sample B, Sample C, Sample Spike-in/AC5) were developed with large numbers of high-confidence positive and negative small variant positions to serve as known content for oncopanel performance assessment. resource
- ★ Eight oncopanels from independent providers were sequenced at ultra-deep depth across multiple independent test laboratories using harmonized protocols. method
- ★ The resulting dataset enables performance assessment of the clinical applicability of oncopanels via sensitivity, false positive rate, and reproducibility metrics. finding
- ★ The dataset supports development and fine-tuning of bioinformatics pipelines for more accurate variant calling across single or multiple panels. resource
- The dataset allows investigation of ideal sequencing depth needed for variant calling at a given minimum VAF and variant type (SNVs, small indels/MNVs, long indels). resource
- The dataset supports evaluation of best use cases for Unique Molecular Identifier (UMI) technology in oncopanel sequencing. resource
- ★ Sample C (1:1 mix of Sample A and Sample B) increases the number of known variants with VAF between 1% and 2.5% by approximately 4-fold relative to Sample A alone. finding
- Sample A contains over 42,000 small variants identified with high confidence across more than 22 million bases of defined regions. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Targeted oncopanel DNA sequencing (capture-based) | Agilent Custom Comprehensive Cancer Panel v2 (AGL), gDNA reference samples A/B/C/Spike-in | none (benchmarking samples) | small variant calls (VCF) | Illumina HiSeq 2500 / NovaSeq 6000 |
| Targeted oncopanel DNA sequencing (capture-based) | Burning Rock DX OncoScreen Plus (BRP), gDNA reference samples | none | small variant calls (VCF) | Illumina NovaSeq 6000 |
| Targeted oncopanel DNA sequencing (capture-based) | Integrated DNA Technologies xGen Pan-Cancer Panel (IDT), gDNA reference samples | none | small variant calls (VCF) | Illumina NovaSeq S4 |
| Targeted oncopanel DNA sequencing (capture-based) | iGeneTech AIOnco-seq (IGT), gDNA reference samples | none | small variant calls (VCF) | Illumina HiSeq 2500 |
| Targeted oncopanel DNA sequencing (capture-based) | Illumina TruSight Tumor 170 (ILM), gDNA reference samples | none | small variant calls (VCF) | Illumina NextSeq v2 |
| Targeted oncopanel DNA sequencing (single primer extension/UMI-based) | QIAGEN Comprehensive Cancer Panel (QGN), gDNA reference samples | none | small variant calls (VCF), UMI-corrected | Illumina NovaSeq |
| Targeted oncopanel DNA sequencing (capture-based) | Roche SeqCap EZ Choice custom PHC Panel (ROC), gDNA reference samples | none | small variant calls (VCF) | Illumina NovaSeq (hg38 reference) |
| Targeted oncopanel DNA sequencing (amplicon-based) | Thermo Fisher Oncomine Comprehensive Assay v3 (TFS), gDNA reference samples | none | small variant calls (VCF) | ThermoFisher IonTorrent S5 |
- – A total of 430 DNA libraries were prepared across eight panels and their recruited testing laboratories. 430 libraries
- – Panel detection limits for variant calling ranged from 0.5% to 2.6% VAF depending on panel. 0.5%-2.6% VAF
- – Sample A contains over 42,000 high-confidence small variants across defined regions. >42,000 variants in >22 Mb
- ▲ Sample C increases known variants in the 1-2.5% VAF range compared to Sample A. 4-fold
- – Three of 28 initially recruited testing laboratories were excluded from performance analysis (affiliation conflict, QC failure, experimental delay). 3 of 28 labs excluded
- count 430 (Total DNA libraries prepared across all panels/labs)
- count 42,000+ (High-confidence small variants identified in Sample A)
- count 22 million bases (Defined high-confidence regions for Sample A variants)
- fold_change 4x (Increase in known variants with VAF 1-2.5% in Sample C vs Sample A)
- count 28 initial labs; 3 excluded (Testing laboratories recruited across 8 panel providers)
- other Detection limits: AGL 1%, BRP 1%, IDT 2%, IGT 1%, ILM 2.6%, QGN 0.5%, ROC 2.5%, TFS 2.5% (Minimum VAF detection limit per oncopanel)
- other Panel sizes (Kbp): AGL 7625, BRP 1631, IDT 780, IGT 944, ILM 527, QGN 837, ROC 149, TFS 349 (Genomic footprint of each oncopanel)
- count 151 members from 100 institutes across 14 countries (Size/composition of the SEQC2 Oncopanel Sequencing Working Group)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a Data Descriptor (not a hypothesis-testing study) describing the design and generation of a multi-laboratory, multi-oncopanel sequencing reference dataset. It reports study design, sample composition, laboratory recruitment/exclusion, and library-preparation protocols for eight oncopanels across technical replicate libraries; the excerpted text does not present formal statistical hypothesis tests, p-values, or summary statistics for performance comparisons.
-
The paper describes reproducibility qualitatively through study design (independent labs, technical replicate libraries, panel providers) rather than presenting a specific quantitative concordance statistic in this excerpt.↳ Could also: An intraclass correlation coefficient (ICC) or Cohen's/Fleiss' kappa across replicate libraries and laboratories — These metrics would numerically summarize agreement/reproducibility across the technical replicates and sites, complementing the descriptive design summary.
-
Variant-calling performance (sensitivity, false positive rate) across panels, laboratories, and VAF levels is discussed conceptually as key metrics.↳ Could also: A mixed-effects (hierarchical) model with laboratory and panel as random effects when analyzing sensitivity/false-positive rate across the nested replicate structure — Because libraries are nested within laboratories which are nested within panels, a mixed-effects approach could separate within-lab technical variability from between-lab and between-panel variability, which a simple pooled comparison would not distinguish.
-
Detection limits (minimum VAF) are reported per panel as point estimates (e.g., 1%, 2%, 2.5%) without an accompanying uncertainty range in the excerpt.↳ Could also: Reporting detection limits alongside a confidence interval or a sensitivity curve (e.g., proportion detected vs. VAF with binomial CIs) — A CI or curve would convey the precision and shape of the detection-limit estimate rather than a single threshold value.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-35680918
Paper: Gong et al. 2022, Sci Data — "Ultra-deep multi-oncopanel sequencing of benchmarking samples with a wide range of variant allele frequencies." (SEQC2 / FDA). This is a data descriptor: it describes a deposited benchmarking dataset rather than making analytic claims. PMID 35680918 · PMC9184574 · DOI 10.1038/s41597-022-01359-6.
Code (assigned): https://github.com/lh3/seqtk — confirmed used in Methods: "FASTQ files were downsampled to an equivalent read count per sample using seqtk v1.0" (IDT read-processing step). seqtk is a generic third-party FASTQ tool; per brief P16, applying it to the paper's own data is a valid reproduction.
Data: SRA BioProject PRJNA677997 (= SRP295113). Open. ~7 TB FASTQ, 504 runs, 5 BioSamples (A, B, C, AC5/Spike-in, AIS), 8 oncopanels, multiple labs. Truth VCFs hosted separately on figshare (19092089, 19128005, collection 5842112.v2).
In scope (pipeline-/deposit-derived, reproduced from metadata)
- Run-count concordance vs Data Records (400 records; A/B/C/AC5 = 103/103/91/103).
- Table 1 "Avg. read count" per panel —
mean(read_count)over the deposited runs per panel; compared 1:1 (with the spots-vs-reads convention) to the printed table. - seqtk equalization — verify the IDT downsampling-to-equal-read-count step is visible in the deposit (identical read counts across IDT runs).
In scope but NOT attempted (honest gaps)
- ~5000X / ~10,000X read depth (Methods): requires aligning reads to each panel's target BED (hg19 for 7 panels, hg38 for ROC) and computing per-base depth — feasible on «our HPC» but beyond the quick clear-data-point target; left for a deeper pass.
- VAF / variant detection-limit validation: requires the figshare truth VCFs + somatic calling per panel; large multi-panel pipeline, out of quick scope.
Out of scope (wet-lab / manual / external)
- Cell-line mixing, DNA extraction, library prep, spike-in formulation, instrument runs.
- Panel vendor specifications (Table 1 left columns) — vendor facts, not computed.
Method
All comparisons derived from the ENA read_run filereport for PRJNA677997
(data/ena_filereport_PRJNA677997.tsv), which reports per-run read_count and
base_count. Per-panel grouping is parsed from the library_name
(Sample<X>_<PANEL><lab>_ST..). A FASTQ-level spot-check (download one run on «infra»,
count reads with seqtk to confirm the spots-vs-reads convention) is pending «our HPC».
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
This Sci Data descriptor reproduces well from the public PRJNA677997 deposit: run counts for the 4 main samples are exact (400=400), the 8 panels and the seqtk v1.0 IDT equalization (all 60 runs = 62,500,000 pairs) are directly confirmed, and 7/8 Table 1 avg read counts match exact or within ~5% after a reads-vs-pairs convention. The two deviations are minor and explainable, sitting on the preprocessing/authors' side: ROC is off 1.15x (486.6M vs 423.0M, likely post-UMI-dedup) and the auxiliary Sample AIS shows 104 vs 103 records (+1). The central claim — that the deposit delivers the promised benchmarking data — holds, so this is a solid partial reproduction with explainable discrepancies rather than a derivability or fabrication concern.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.