A crowdsourced set of curated structural variants for the human genome.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓The central claim held under reproduction
- 🟡Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
PMID 32559231 (Chapman et al., PLoS Comput Biol 2020) is a CROWDSOURCED HUMAN-CURATION study of 1,235 structural variants in GIAB HG002 via the SVCurator web app. Its headline numbers about curator behaviour (93% expert concordance, 37/61 curators, time-to-curate) are human-generated and OUT of computational scope. Described well enough? Mixed: the deposited supplementary tables (S1 Data=s008, S2 Table=s007) fully support the EXACT recomputation of C1 (935 events = 496 insertions + 439 deletions) and the minimum SV size (20 bp). BUT the paper's other headline denominators are NOT regenerable from deposited data because they were computed over larger INTERMEDIATE sets that were not deposited: C2's 698 'sites inside v0.6 regions' (top-curator labels over the full pre-filter cohort; the deposited 935 final set gives 627/635=98.7%, not 669/698=94.5%), C3's 879 (only the 75-event <2-tech complement is deposited, which is internally consistent with 804/879), and C4's 892,149 bp max (the 1295 evaluated set, vs 816,127 in the deposited 935). One paper-internal arithmetic inconsistency was found: 669/698 = 95.84%, not the stated 94.5% (consistent only at denominator ~708). DEEPER (D1): the paper's actual tool svviz2 v2.0a3 (commit 163bca0d) was BUILT FROM SOURCE on «our HPC» (7 build fixes incl. gcc-14 -fcommon, Ubuntu conda-sysroot ld, scipy.misc.comb) and re-genotyped 2 curated deletions DIRECTLY from the GIAB HG002 PacBio-70x + 10x-84x BAMs (reads remote-sliced over https). For both events svviz2 calls GT 0/1 (heterozygous) in BOTH technologies, reproducing the curators' Het_Var labels AND the >=2-technology support behind C3 -- a genuine end-to-end pipeline reproduction on the paper's own data. NOT attempted: the human crowdsourcing/curation itself (impossible to reproduce without 61 human curators); the full 804/879 and 669/698 over all events (intermediate label sets not deposited; svviz2 parameters undocumented); SVanalyzer merging; IGV imaging. Honest outcome: PARTIAL -- C1 exact, D1 mechanism reproduced on real data, with clear evidence of which reported values cannot be checked against deposited materials (no fabrication concluded; the gaps are non-deposited intermediates plus one arithmetic typo).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 71assessed: 2026-06-17 ⛓ 2d549cece465
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-17
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-18no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan crowdsourced manual curation of structural variants (SVs)—using a web app that displays sequencing evidence from multiple technologies—reliably evaluate putative large indels and SVs in the human genome to produce high-confidence labels for SV type, size, and genotype?
- ★ A crowdsourcing web application (SVCurator) enables curators to manually review and label large indels and SVs by displaying short, long, and linked read sequencing evidence from GIAB HG002. resource
- ★ Manual curation produced high-confidence consensus labels for 935 SV events after filtering low-concordance events from 1235 curated SVs. finding
- ★ SVCurator crowdsourced labels were 94.5% concordant with the heuristic-based GIAB v0.6 draft benchmark SV callset. finding
- ★ Curators can successfully evaluate putative SVs (including insertions, not just deletions) when given evidence from multiple sequencing technologies via svviz2/IGV visualizations. method
- ★ 'Expert' curators were 93% concordant with each other, and 37 of 61 curators reached at least 78% concordance with experts. finding
- ★ Curators were least concordant for complex SVs and SVs with inaccurate breakpoints or size predictions. finding
- svviz2 haplotype-partitioned images allow visualization of reads aligned to reference vs alternate alleles for both deletions and insertions. method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Illumina 250bp paired-end whole genome sequencing | GIAB Ashkenazi Jewish Trio son HG002/NA24385 [NIST RM 8391] | none | reads aligned to reference/alternate alleles for SV visualization (svviz2/IGV) | Illumina |
| Illumina 150bp paired-end whole genome sequencing | GIAB HG002/NA24385 | none | read alignment evidence for SVs | Illumina |
| Illumina 6kb mate-pair sequencing | GIAB HG002/NA24385 | none | read alignment evidence for SVs | Illumina |
| Haplotype-partitioned long-read whole genome sequencing | GIAB HG002/NA24385 | none | haplotype-partitioned reads aligned to reference/alternate alleles | PacBio |
| Haplotype-partitioned linked-read whole genome sequencing | GIAB HG002/NA24385 | none | haplotype-partitioned reads aligned to reference/alternate alleles | 10x Genomics |
| Manual crowdsourced curation (SVCurator web app) | GIAB HG002/NA24385 candidate SV callset | none | labels for SV type (deletion/insertion), size accuracy (within ±20%), and genotype with confidence score | SVCurator Python web app; svviz2; IGV |
- – 'Expert' curators were concordant with each other on labels assigned to each event 93%
- – Top curators' labels concordant with GIAB v0.6 benchmark genotype labels (669 of 698 curated sites inside benchmark regions) 94.5%
- – High-confidence final labels produced after filtering low-concordance events 935 events
- – Deletions averaged higher concordance than insertions among experts (86% vs 80%) 86% vs 80%
- ▼ Complex events had lowest concordance among top curators (mean 64% Threshold 1, 47% Threshold 2) 64% / 47%
- – 20 curators evaluating >648 events had concordance with expert consensus labels 87%
- – 1162 events (527 deletions, 635 insertions) retained after filtering 133 sites with discordant threshold consensus 1162 events
- – Expert concordance with consensus label ranged across individuals; 100% concordant for 407 events 77.7%-100%
- count 1235 SVs manually curated (total curated SVs producing labels)
- count 1295 SV calls (579 deletions and 716 insertions) (events evaluated in SVCurator)
- other 93% (average concordance among 'expert' curators)
- other 94.5% (669/698) (top curator labels concordant with v0.6 benchmark inside benchmark regions)
- count 935 events assigned final high-confidence labels (events with >=60% top-curator concordance and >=3 agreeing)
- count 61 of 136 registered participants evaluated events (curator participation)
- mean 47.31 seconds average curation time per event (time to curate each event)
- count 29 events discordant between curators and v0.6; 11 labeled complex (discordant events analysis)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This crowdsourcing study had 61 curators (including 7 designated experts) evaluate 1295 putative structural variant (SV) calls from the GIAB HG002 genome using the SVCurator web platform. The primary analytical approach was descriptive: expert consensus labels were determined by simple majority voting, each curator's quality was assessed by percent concordance with expert consensus on a reference set of 541 events, and concordance-based thresholds were used to filter curators and events. No formal inferential hypothesis tests were applied; results were reported as percent agreement and summary counts, with distributions displayed as box plots.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Percent concordance (proportion agreement) | Concordance of each of 61 curators with expert consensus label across 541 reference events; overall expert-to-expert concordance; SVCurator labels vs. GIAB v0.6 benchmark (698 events) | 541 events with ≥68% expert concordance and ≥4 expert agreement used as reference set; 698 events inside v0.6 benchmark regions for label comparison | not stated |
| Simple majority (plurality) voting | Determination of expert consensus genotype label for each SV event | 7 expert curators per event | na |
| Leave-one-out concordance scoring | Scoring each of the 7 expert curators against consensus of the remaining experts, to derive concordance thresholds for screening all curators | 7 expert curators evaluated against events with ≥68% concordance and ≥3 expert agreement | not stated |
| Fixed threshold-based curator filtering | Screening all 61 curators into Threshold 1 (≥90.9% concordance) and Threshold 2 (≥77.7% concordance) groups, anchored to the two lowest-performing expert scores | 61 curators compared against 541 reference events | not stated |
-
Inter-rater agreement was quantified as percent concordance (raw proportion agreement) between curators and the expert consensus label↳ Could also: Fleiss' kappa or Krippendorff's alpha could also be used to measure inter-rater reliability across multiple raters on a multi-class categorical outcome — Chance-corrected agreement statistics account for the baseline probability of agreement by chance, which is especially relevant when some genotype categories (e.g., heterozygous variant) are substantially more common than others, making raw percent agreement potentially optimistic
-
Expert consensus labels were determined by simple majority (plurality) voting among 7 expert curators, with all experts weighted equally↳ Could also: Probabilistic aggregation models such as the Dawid-Skene model or MACE (Multi-Annotator Competence Estimation) could also be used to infer latent true labels while simultaneously estimating per-annotator reliability — Model-based aggregation weights annotators by their inferred reliability and estimates ground-truth labels jointly, which can be advantageous when annotator skill is heterogeneous or the number of annotators per item is small and unequal
-
Curator quality thresholds (90.9% and 77.7%) were fixed at the concordance scores of the second-lowest and lowest expert curators, respectively↳ Could also: Data-driven thresholds based on the empirical distribution of all curators' concordance scores (e.g., a natural gap in the distribution, a mixture model, or a percentile cutoff) could also be applied — Anchoring thresholds to the specific experts chosen is intuitive and transparent, but a distribution-based approach would make the cutoff less sensitive to the particular composition of the expert panel and could generalize more readily to future curation efforts
-
The concordance between SVCurator final labels and GIAB v0.6 benchmark genotypes was reported as a single overall proportion (94.5% of 698 events) without a measure of precision↳ Could also: A 95% Wilson score confidence interval around the concordance proportion could also be reported — A confidence interval conveys the statistical precision of the concordance estimate given the sample size (n=698), allowing readers to gauge how much the observed concordance might vary across different curated event sets
-
Concordance distributions across SV types (deletions vs. insertions) and curator groups were described verbally and displayed as box plots, without formal comparative tests↳ Could also: A Wilcoxon rank-sum test or permutation test comparing concordance distributions between groups (e.g., deletions at 86% vs. insertions at 80% among experts) could also be applied — A formal test would allow readers to assess whether observed differences in concordance between groups are larger than expected by chance given the number of events and curators, rather than relying solely on visual comparison
-
SVs were randomly sampled within each of 7 pre-defined size bins without a stated sample size justification↳ Could also: A prospective power analysis or sample size calculation targeting a specified precision for the concordance estimate (e.g., a margin of error of ±5%) could also be used to determine the number of events and curations per event — A formal sample size rationale would make the sampling scheme more explicit and reproducible, and would help clarify whether the study had adequate events in rarer or more challenging categories such as large insertions and complex SVs
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-32559231
Paper: Chapman et al. (2020) "A crowdsourced set of curated structural variants for the human genome." PLoS Comput Biol 16(6):e1007933.
Nature of the paper. The headline contribution is a crowdsourced human curation of 1,235 putative structural variants (SVs) in the GIAB Ashkenazi son HG002, using the SVCurator web app. Most headline numbers are human-derived (curator concordance, agreement among 61 curators, time-to-curate) and therefore cannot be reproduced computationally — they would require re-running a crowd of 61 human curators. Those are OUT of scope.
However, several reported results are pipeline-derived / deterministic recomputations from shipped data and ARE in scope:
IN SCOPE (pipeline-derived, shipped-data recomputations)
| id | reported | where | how computed |
|---|---|---|---|
| C1 | 935 high-confidence labeled events (496 insertions, 439 deletions) | Abstract / Results; S1 Data | row count + SVTYPE breakdown of S1 Data |
| C2 | 94.5% concordance with GIAB v0.6 benchmark = 669/698 curated sites inside v0.6 regions matching v0.6 genotype | Results; Fig 5 text | join curator labels with v0.6 genotypes, restrict to v0.6 regions, fraction matching |
| C3 | 92.2% of events supported by ≥2 sequencing technologies = 804/879 (from svviz2 genotypes) | Results | per-technology svviz2 genotype columns → count events with ≥2 supporting techs |
| C4 | size range 20–892,149 bp | Results | min/max of SV size in shipped labels |
Tools named: svviz2 v2.0a3 (haplotype-partitioned genotyping/imaging, github.com/nspies/svviz2), SVanalyzer (sequence-resolved merging, ≥20% similarity), IGV (imaging only).
DEEPER (in scope, harder) — re-run the tool, not just recompute
- D1: Build svviz2 v2.0a3 and re-genotype a subset of the curated SVs against the GIAB HG002 BAMs (Illumina 250/150/MP, PacBio, 10x) listed in Table 1; compare the regenerated per-technology genotypes to the shipped svviz2 genotypes (S2 Table / S2 Data). This is the genuine pipeline reproduction beyond the arithmetic recomputation.
OUT OF SCOPE (human / manual — not attempted)
- 93% expert-curator concordance; individual expert range 77.7–100%.
- 37 of 61 curators ≥78% concordant with experts; Top-Curator thresholds.
- Manual re-curation results (S2 Table human column).
- SVCurator web-app deployment, IGV image inspection, time-to-curate. These require human curators and cannot be reproduced computationally.
Data / code pointers
- Code: https://github.com/nspies/svviz2 (svviz2 2.0a3); SVanalyzer https://github.com/nhansen/SVanalyzer
- Supplementary (PLOS, journal.pcbi.1007933.s006..s011): S1 Table=s006, S2 Table=s007, S1 Data=s008, S2 Data=s009, S3 Data=s010, S4 Data=s011.
- Sequencing data: SRA PRJNA200694; GIAB FTP union callset ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/AshkenazimTrio/analysis/NIST_UnionSVs_12122017/SVmerge121217/ (union_171212_refalt.sort.vcf.gz) and v0.6 benchmark.
- «infra» work dir: «path»
Strategy
- (Quick minimum) Recompute C1–C4 from the shipped supplementary tables — deterministic, this is the auditability check: does the shipped data actually yield the reported numbers?
- (Deeper) D1: re-run svviz2 on a subset against the GIAB BAMs.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
Partial but credible reproduction. C1 reproduces exactly from deposited S1 Data (935 = 496 insertions + 439 deletions; min 20 bp), and the deeper D1 check rebuilt svviz2 v2.0a3 from source and reproduced the curators' Het_Var calls and >=2-tech support on the paper's own GIAB BAMs. The deviations (C2 94.5% vs 98.7%; C3 full 804/879; C4 892,149 vs 816,127 bp) sit on the authors'/data-availability side — the reported headline denominators come from intermediate sets that were never deposited, compounded by one paper-internal arithmetic typo (669/698=95.84%, not 94.5%). Severity is moderate (magnitude and direction hold, concordance is if anything higher in the deposited set) and there is no fabrication signal — the deposited 75-event <2-tech complement is exactly internally consistent with 879-804. Overall yellow: the central conclusion holds, with deviations cleanly attributable to missing intermediates rather than computational error.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.