A crowdsourced set of curated structural variants for the human genome.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓The central claim held under reproduction
- 🟡Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
PMID 32559231 (Chapman et al., PLoS Comput Biol 2020) is a CROWDSOURCED HUMAN-CURATION study of 1,235 structural variants in GIAB HG002 via the SVCurator web app. Its headline numbers about curator behaviour (93% expert concordance, 37/61 curators, time-to-curate) are human-generated and OUT of computational scope. Described well enough? Mixed: the deposited supplementary tables (S1 Data=s008, S2 Table=s007) fully support the EXACT recomputation of C1 (935 events = 496 insertions + 439 deletions) and the minimum SV size (20 bp). BUT the paper's other headline denominators are NOT regenerable from deposited data because they were computed over larger INTERMEDIATE sets that were not deposited: C2's 698 'sites inside v0.6 regions' (top-curator labels over the full pre-filter cohort; the deposited 935 final set gives 627/635=98.7%, not 669/698=94.5%), C3's 879 (only the 75-event <2-tech complement is deposited, which is internally consistent with 804/879), and C4's 892,149 bp max (the 1295 evaluated set, vs 816,127 in the deposited 935). One paper-internal arithmetic inconsistency was found: 669/698 = 95.84%, not the stated 94.5% (consistent only at denominator ~708). DEEPER (D1): the paper's actual tool svviz2 v2.0a3 (commit 163bca0d) was BUILT FROM SOURCE on «our HPC» (7 build fixes incl. gcc-14 -fcommon, Ubuntu conda-sysroot ld, scipy.misc.comb) and re-genotyped 2 curated deletions DIRECTLY from the GIAB HG002 PacBio-70x + 10x-84x BAMs (reads remote-sliced over https). For both events svviz2 calls GT 0/1 (heterozygous) in BOTH technologies, reproducing the curators' Het_Var labels AND the >=2-technology support behind C3 -- a genuine end-to-end pipeline reproduction on the paper's own data. NOT attempted: the human crowdsourcing/curation itself (impossible to reproduce without 61 human curators); the full 804/879 and 669/698 over all events (intermediate label sets not deposited; svviz2 parameters undocumented); SVanalyzer merging; IGV imaging. Honest outcome: PARTIAL -- C1 exact, D1 mechanism reproduced on real data, with clear evidence of which reported values cannot be checked against deposited materials (no fabrication concluded; the gaps are non-deposited intermediates plus one arithmetic typo).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 71assessed: 2026-06-17 ⛓ 2d549cece465
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-17
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-18no human curator yet
- Last updated
- 2026-09-19
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetManual crowdsourced curation of putative structural variants, when curators are shown multi-technology sequencing evidence, can reliably distinguish true SVs from false positives and assign accurate type/size/genotype labels suitable for benchmarking SV callsets.
- ★ 1235 manually curated SVs were produced that can be used to evaluate SV callers or train machine learning models resource
- ★ SVCurator, a crowdsourcing web app, was developed to let curators review large indels and SVs using images from short, long, and linked-read sequencing and report genotype/size accuracy method
- 'Expert' curators were 93% concordant with each other on assigned labels finding
- 37 of 61 curators had at least 78% concordance with the expert curator set finding
- ★ After filtering events with low curator concordance, high confidence labels were produced for 935 events finding
- ★ SVCurator crowdsourced labels were 94.5% concordant with the heuristic-based draft GIAB v0.6 benchmark SV callset finding
- Curators were least concordant for complex SVs and for SVs with inaccurate breakpoints or size predictions finding
- ★ Curators can successfully evaluate putative SVs when given evidence from multiple sequencing technologies finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| crowdsourced manual curation (SVCurator web app) | HG002 / NA24385, GIAB Ashkenazi Jewish Trio son | none | SV type (deletion/insertion), size accuracy within 20%, genotype label, genotype confidence score | SVCurator Python web application |
| short-read whole genome sequencing, Illumina 250bp paired-end | HG002 / NA24385 | none | read alignment evidence supporting reference or alternate allele | Illumina |
| short-read whole genome sequencing, Illumina 150bp paired-end | HG002 / NA24385 | none | read alignment evidence supporting reference or alternate allele | Illumina |
| mate-pair sequencing, Illumina 6kb | HG002 / NA24385 | none | read alignment evidence supporting reference or alternate allele | Illumina |
| long-read sequencing, haplotype-partitioned PacBio | HG002 / NA24385 | none | haplotype-resolved read alignment evidence | PacBio |
| linked-read sequencing, haplotype-partitioned 10x Genomics | HG002 / NA24385 | none | haplotype-resolved read alignment evidence | 10x Genomics |
| read/dotplot image visualization for curator review | HG002 sequencing reads across technologies | none | visualized alignments and repeat-region dotplots used as curation evidence | svviz2, IGV (Integrative Genomics Viewer) |
| benchmark comparison against sequence-resolved SV callset | HG002 / NA24385 | none | genotype concordance between curator labels and benchmark labels | — |
- – Curators evaluated 1295 SV calls (579 deletions and 716 insertions) sampled across 7 size bins 1295 events (579 deletions, 716 insertions)
- – The 7 expert curators were concordant with each other overall, with 100% concordance on 407 events 93% overall; 100% on 407 events; range 77.7%-100% per expert
- – Deletions had higher expert concordance than insertions 86% (deletions) vs 80% (insertions)
- – Top curators were split into two concordance-threshold groups 26 curators ≥90.9% (Threshold 1); 37 curators ≥77.7% (Threshold 2)
- – After filtering, 1162 events (527 deletions, 635 insertions) were retained; final high-confidence labels assigned to a subset 935 events with ≥60% top-curator concordance and ≥3 agreeing curators
- – SVCurator labels matched the GIAB v0.6 benchmark genotype labels for most curated sites within benchmark regions 669/698 (94.5%) concordant
- ▼ Complex SVs had the lowest concordance among top curators in both threshold groups mean 64% (Threshold 1) and 47% (Threshold 2) concordance
- – 20 high-volume curators (evaluating >648 events) showed strong agreement with expert consensus 87% average concordance
- count 1235 curated SVs (final curated SV set produced for benchmarking/ML training)
- other 93% concordance among expert curators (agreement among 7 GIAB expert curators)
- other 94.5% concordance (669/698) (top-curator labels vs GIAB v0.6 benchmark within benchmark regions)
- count 61 of 136 registered participants evaluated events (SVCurator app participation)
- mean mean of 11 curators evaluated each event (events curated at least 3 times)
- mean average curation time 47.31 seconds per event (range <10s to >120s across curators)
- count 935 events assigned high-confidence final labels (after filtering low-concordance curator responses)
- other expert concordance range 77.7% to 100% (leave-one-out concordance scores used to set Threshold 1 (90.9%) and Threshold 2 (77.7%))
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This crowdsourcing study had 61 curators (including 7 designated experts) evaluate 1295 putative structural variant (SV) calls from the GIAB HG002 genome using the SVCurator web platform. The primary analytical approach was descriptive: expert consensus labels were determined by simple majority voting, each curator's quality was assessed by percent concordance with expert consensus on a reference set of 541 events, and concordance-based thresholds were used to filter curators and events. No formal inferential hypothesis tests were applied; results were reported as percent agreement and summary counts, with distributions displayed as box plots.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Percent concordance (proportion agreement) | Concordance of each of 61 curators with expert consensus label across 541 reference events; overall expert-to-expert concordance; SVCurator labels vs. GIAB v0.6 benchmark (698 events) | 541 events with ≥68% expert concordance and ≥4 expert agreement used as reference set; 698 events inside v0.6 benchmark regions for label comparison | not stated |
| Simple majority (plurality) voting | Determination of expert consensus genotype label for each SV event | 7 expert curators per event | na |
| Leave-one-out concordance scoring | Scoring each of the 7 expert curators against consensus of the remaining experts, to derive concordance thresholds for screening all curators | 7 expert curators evaluated against events with ≥68% concordance and ≥3 expert agreement | not stated |
| Fixed threshold-based curator filtering | Screening all 61 curators into Threshold 1 (≥90.9% concordance) and Threshold 2 (≥77.7% concordance) groups, anchored to the two lowest-performing expert scores | 61 curators compared against 541 reference events | not stated |
-
Inter-rater agreement was quantified as percent concordance (raw proportion agreement) between curators and the expert consensus label↳ Could also: Fleiss' kappa or Krippendorff's alpha could also be used to measure inter-rater reliability across multiple raters on a multi-class categorical outcome — Chance-corrected agreement statistics account for the baseline probability of agreement by chance, which is especially relevant when some genotype categories (e.g., heterozygous variant) are substantially more common than others, making raw percent agreement potentially optimistic
-
Expert consensus labels were determined by simple majority (plurality) voting among 7 expert curators, with all experts weighted equally↳ Could also: Probabilistic aggregation models such as the Dawid-Skene model or MACE (Multi-Annotator Competence Estimation) could also be used to infer latent true labels while simultaneously estimating per-annotator reliability — Model-based aggregation weights annotators by their inferred reliability and estimates ground-truth labels jointly, which can be advantageous when annotator skill is heterogeneous or the number of annotators per item is small and unequal
-
Curator quality thresholds (90.9% and 77.7%) were fixed at the concordance scores of the second-lowest and lowest expert curators, respectively↳ Could also: Data-driven thresholds based on the empirical distribution of all curators' concordance scores (e.g., a natural gap in the distribution, a mixture model, or a percentile cutoff) could also be applied — Anchoring thresholds to the specific experts chosen is intuitive and transparent, but a distribution-based approach would make the cutoff less sensitive to the particular composition of the expert panel and could generalize more readily to future curation efforts
-
The concordance between SVCurator final labels and GIAB v0.6 benchmark genotypes was reported as a single overall proportion (94.5% of 698 events) without a measure of precision↳ Could also: A 95% Wilson score confidence interval around the concordance proportion could also be reported — A confidence interval conveys the statistical precision of the concordance estimate given the sample size (n=698), allowing readers to gauge how much the observed concordance might vary across different curated event sets
-
Concordance distributions across SV types (deletions vs. insertions) and curator groups were described verbally and displayed as box plots, without formal comparative tests↳ Could also: A Wilcoxon rank-sum test or permutation test comparing concordance distributions between groups (e.g., deletions at 86% vs. insertions at 80% among experts) could also be applied — A formal test would allow readers to assess whether observed differences in concordance between groups are larger than expected by chance given the number of events and curators, rather than relying solely on visual comparison
-
SVs were randomly sampled within each of 7 pre-defined size bins without a stated sample size justification↳ Could also: A prospective power analysis or sample size calculation targeting a specified precision for the concordance estimate (e.g., a margin of error of ±5%) could also be used to determine the number of events and curations per event — A formal sample size rationale would make the sampling scheme more explicit and reproducible, and would help clarify whether the study had adequate events in rarer or more challenging categories such as large insertions and complex SVs
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-32559231
Paper: Chapman et al. (2020) "A crowdsourced set of curated structural variants for the human genome." PLoS Comput Biol 16(6):e1007933.
Nature of the paper. The headline contribution is a crowdsourced human curation of 1,235 putative structural variants (SVs) in the GIAB Ashkenazi son HG002, using the SVCurator web app. Most headline numbers are human-derived (curator concordance, agreement among 61 curators, time-to-curate) and therefore cannot be reproduced computationally — they would require re-running a crowd of 61 human curators. Those are OUT of scope.
However, several reported results are pipeline-derived / deterministic recomputations from shipped data and ARE in scope:
IN SCOPE (pipeline-derived, shipped-data recomputations)
| id | reported | where | how computed |
|---|---|---|---|
| C1 | 935 high-confidence labeled events (496 insertions, 439 deletions) | Abstract / Results; S1 Data | row count + SVTYPE breakdown of S1 Data |
| C2 | 94.5% concordance with GIAB v0.6 benchmark = 669/698 curated sites inside v0.6 regions matching v0.6 genotype | Results; Fig 5 text | join curator labels with v0.6 genotypes, restrict to v0.6 regions, fraction matching |
| C3 | 92.2% of events supported by ≥2 sequencing technologies = 804/879 (from svviz2 genotypes) | Results | per-technology svviz2 genotype columns → count events with ≥2 supporting techs |
| C4 | size range 20–892,149 bp | Results | min/max of SV size in shipped labels |
Tools named: svviz2 v2.0a3 (haplotype-partitioned genotyping/imaging, github.com/nspies/svviz2), SVanalyzer (sequence-resolved merging, ≥20% similarity), IGV (imaging only).
DEEPER (in scope, harder) — re-run the tool, not just recompute
- D1: Build svviz2 v2.0a3 and re-genotype a subset of the curated SVs against the GIAB HG002 BAMs (Illumina 250/150/MP, PacBio, 10x) listed in Table 1; compare the regenerated per-technology genotypes to the shipped svviz2 genotypes (S2 Table / S2 Data). This is the genuine pipeline reproduction beyond the arithmetic recomputation.
OUT OF SCOPE (human / manual — not attempted)
- 93% expert-curator concordance; individual expert range 77.7–100%.
- 37 of 61 curators ≥78% concordant with experts; Top-Curator thresholds.
- Manual re-curation results (S2 Table human column).
- SVCurator web-app deployment, IGV image inspection, time-to-curate. These require human curators and cannot be reproduced computationally.
Data / code pointers
- Code: https://github.com/nspies/svviz2 (svviz2 2.0a3); SVanalyzer https://github.com/nhansen/SVanalyzer
- Supplementary (PLOS, journal.pcbi.1007933.s006..s011): S1 Table=s006, S2 Table=s007, S1 Data=s008, S2 Data=s009, S3 Data=s010, S4 Data=s011.
- Sequencing data: SRA PRJNA200694; GIAB FTP union callset ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/AshkenazimTrio/analysis/NIST_UnionSVs_12122017/SVmerge121217/ (union_171212_refalt.sort.vcf.gz) and v0.6 benchmark.
- «infra» work dir: «path»
Strategy
- (Quick minimum) Recompute C1–C4 from the shipped supplementary tables — deterministic, this is the auditability check: does the shipped data actually yield the reported numbers?
- (Deeper) D1: re-run svviz2 on a subset against the GIAB BAMs.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
Partial but credible reproduction. C1 reproduces exactly from deposited S1 Data (935 = 496 insertions + 439 deletions; min 20 bp), and the deeper D1 check rebuilt svviz2 v2.0a3 from source and reproduced the curators' Het_Var calls and >=2-tech support on the paper's own GIAB BAMs. The deviations (C2 94.5% vs 98.7%; C3 full 804/879; C4 892,149 vs 816,127 bp) sit on the authors'/data-availability side — the reported headline denominators come from intermediate sets that were never deposited, compounded by one paper-internal arithmetic typo (669/698=95.84%, not 94.5%). Severity is moderate (magnitude and direction hold, concordance is if anything higher in the deposited set) and there is no fabrication signal — the deposited 75-event <2-tech complement is exactly internally consistent with 879-804. Overall yellow: the central conclusion holds, with deviations cleanly attributable to missing intermediates rather than computational error.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at [email protected].
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.