The third international hackathon for applying insights into large-scale genomic composition to use cases in a wide range of organisms.
The main results reproduced, with only marginal, non-material deviations.
- ✓No relevant deviation in data/preprocessing
- ✓Any deviation was negligible
- 🟡Could not use the authors’ exact input data
- 🔴Reported values were only indirectly comparable
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Hackathon report (F1000Research). The in-scope reproducible artifact is SimiSTR (third-party STR-injection simulator, P16) that generates the STRdust test data. We reproduced it on «our HPC» (SLURM «job») on the repo's shipped hg38-chr22 Testcase: SimiSTR runs and emits a modified diploid FASTA + a ground-truth change BED exactly as described -- 7/7 functional checks pass (diploid doubling, per-allele edits, expansion within the -ma bound, and 16 Mbp outside STR regions byte-identical proving edits are local). The -snv substitution rate reproduces within tolerance at two rates on a synthetic 50 kb STR (0.00116 vs 0.001; 0.01012 vs 0.01). SimiSTR is UNSEEDED (random.* + numpy.random with no seed), so shipped example outputs are NOT byte-reproducible by design -- reproduction is functional + statistical, never byte-exact; run1 vs run2 differ as expected. One minor tool quirk recorded: for sub-2-repeat STRs the boundary-finder's +/-9 bp shift can put an expansion off the motif frame (TCTTT got +16 not a multiple of 5); the byte-bound still holds. NOT attempted: (a) the paper's STRdust-vs-Straglr-vs-TRiCoLOR accuracy comparison -- it reports NO numbers (no_expected_result); (b) the tomato SL4.0 arm -- no tomato input bundle is shipped. PRJEB5235 (84 tomato WGS short-read runs) is an incidentally-harvested accession the long-read SimiSTR/STRdust workflow does not consume (wrong modality). One code change: a 1-line Biopython>=1.78 API-compat shim (guarded FastaWriter.write_header, a no-op for FASTA); STR logic untouched. All grades provisional -- a human signs off.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-19 ⛓ c4768f38f7b5
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-29
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe hackathon's central goal was to test whether prototyping open-source software during a short collaborative hackathon could advance structural variant (SV) research across diverse organisms, including SV discovery, genotyping, sequence reconstruction, and clinically relevant SVs such as SARS-CoV-2 variants.
- ★ The third BCM & DNAnexus hackathon produced nine software projects for SV analysis, of which eight are presented in this paper. finding
- ★ STRdust is a new tool to accurately detect and genotype short tandem repeats (STRs) from long-read sequencing data. method
- ★ kTom characterizes k-mer content of re-sequenced genomes to identify k-mers unique to trait-introgressed samples, enabling reference-free tracking of tomato trait introgressions. method
- ★ INSeption aims to polish/resolve structural variant insertions. method
- Existing long-read STR calling tools (PacmonSTR, NanoSatellite, TRiCoLOR, Straglr) have limited usability due to platform and/or computational demands, motivating development of STRdust. finding
- Assembly-based SV calling approaches circumvent reference-sequence biases inherent to alignment-based approaches, whereas alignment-based approaches poorly resolve complex or large rearrangements/insertions exceeding read length. mechanism
- Most current tomato trait marker sets rely exclusively on SNPs, which do not always adequately track diverse tomato genetics. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Long-read STR detection and genotyping | long-read sequenced genomes (human and other organisms) | none | STR genotype/length calls | PacBio Sequel II System; Oxford Nanopore Technologies (ONT) |
| k-mer profiling of re-sequenced genomes | re-sequenced tomato accessions | trait introgression from wild relatives | k-mers unique to traited samples | — |
| Read-based SV calling (alignment- and assembly-based) for insertion polishing | genomic sequencing reads | structural variant insertions | resolved/polished SV insertion sequences | BWA, Minimap2, NGMLR, DELLY, SNIFFLES, Canu, Flye, BlasR, SGVar, Paftools |
- – Hackathon convened 59 scientists from 14 countries and 13 U.S. states to develop SV-focused software
- – PacBio HiFi long reads have an error rate of approximately 1% ~1%
- – ONT long reads have an error rate of approximately 10-15% ~10-15%
- – Global tomato production exceeded 180 million tons in 2019 >180 million tons
- count 59 scientists (hackathon participants from 14 countries and 13 U.S. states)
- count 14 countries (countries represented by hackathon participants)
- count nine hackathon projects (eight presented) (number of software projects developed at the hackathon)
- other >180 million tons (global tomato production in 2019 (FAOSTAT))
- other ~1% (PacBio HiFi read error rate)
- other ~10-15% (ONT read error rate)
- other ≥50 bp (minimum size threshold defining structural variants (SVs))
- other 2-6 bp (short tandem repeat (STR) motif length)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a Software Tool Article reporting on outputs of a genomics hackathon: eight bioinformatics software projects for structural variant (SV) detection, genotyping, and characterization (e.g., STRdust, kTom, INSeption, GeneVar2, cov2db, K-var, Imavirus, and a Reference Panel Generator). The text provided describes tool design, motivation, and use cases rather than a hypothesis-driven experiment with defined comparison groups; no formal inferential statistical tests, p-values, or effect-size reporting appear in the excerpt supplied.
-
The paper reports approximate error rates for PacBio HiFi (~1%) and ONT (~10-15%) long-read platforms as background context rather than as measured outcomes with variance estimates↳ Could also: Presenting these rates with an accompanying range, confidence interval, or citation-specific dataset detail — When platform error rates are used to motivate a tool's design, showing the underlying variability or source dataset can help readers judge how the tool's performance might generalize across sequencing runs or chemistries
-
The described tools (e.g., STRdust, kTom, INSeption) are introduced with qualitative rationale for their approach compared to prior tools (e.g., PacmonSTR, NanoSatellite, TRiCoLOR, Straglr for STR calling)↳ Could also: A quantitative benchmarking comparison (e.g., precision/recall, F1 score, or runtime comparisons with confidence intervals) against these existing tools on a shared benchmark dataset — Formal benchmarking with summary statistics would allow readers to quantitatively compare the new tool's performance to established alternatives, complementing the qualitative rationale given
-
The kTom project uses re-sequenced tomato accessions to demonstrate k-mer-based introgression detection without a stated sample size or replication scheme in the excerpt↳ Could also: Reporting the number of accessions/individuals analyzed and whether biological or technical replicates were used — Stating sample size and replicate structure helps readers assess the generalizability of the k-mer profiling approach across tomato populations
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-36262335
Paper: Walker K, Kalra D, Lowdon R, ... Sedlazeck FJ, Busby B. The third international hackathon for applying insights into large-scale genomic composition to use cases in a wide range of organisms. F1000Research 2022, 11:1126. PMID 36262335 · PMCID PMC9557141 · DOI 10.12688/f1000research.110194.1
Nature of the paper
This is a hackathon report: it bundles ~10 independent mini-projects (teams) into one F1000Research paper. It is descriptive — it narrates what each team built during a 3-day hackathon. The body reports essentially no quantitative benchmark tables (no precision/recall/runtime numbers per tool). The reproduction unit was harvested from the SimiSTR code link + a tomato SRA accession.
The relevant sub-project: "STRdust" (Detect and genotype short tandem repeats)
- STRdust: a new long-read STR caller that parses CIGAR strings to find large
insertions / soft-clips, builds a consensus, then runs
mreps 6.2.01to find the repeat unit. Prep: align with Minimap2 2.24 (±-Y), phase with longshot 0.4.1. - SimiSTR (the harvested repo, https://github.com/DamarisLa/SimiSTR): a Python tool that manipulates a reference genome to inject ground-truth STR variation (expansion/contraction, SNVs, single-base indels; haploid or diploid). Used to build the test datasets STRdust was evaluated on.
- Paper's exact words on the test data: "STRdust was tested against simulated STR datasets produced by SimiSTR. SimiSTR modified the GRCh38 (human) and SL4.0 (tomato) reference genome assemblies. Additional variation (SNVs) was introduced with SURVIVOR 1.0.7 at a rate of 0.001. Long reads were simulated using SURVIVOR ... Mapping ... Minimap2 2.24 ... phasing ... longshot 0.4.1. STRdust results were compared to TRiCoLOR 1.1 and Straglr 1.1.1 using default parameters."
In scope (pipeline-derived, attemptable)
| id | result | pipeline | attemptable? |
|---|---|---|---|
| C1 | SimiSTR runs on a reference assembly + STR BED and emits a modified FASTA + a ground-truth change BED (expansion/SNV/indel), as described | SimiSTR_v8.py (Python 3.8 + Biopython + numpy) | YES — repo ships a complete Testcase (hg38 chr22 + input/expected-output BED + expected output FASTA). P16: third-party-tool-on-paper's-data is valid. |
| C2 | The diploid output doubles each modified chromosome into two alleles and the change-BED records per-allele original→new repeat counts | SimiSTR_v8.py -dip 2 |
YES |
| C3 | Modifications are bounded (added repeats ≤ -ma, removed ≤ STR length); SNVs occur at ~the -snv rate |
SimiSTR_v8.py | YES — statistical check |
Out of scope / not attemptable
- STRdust vs Straglr vs TRiCoLOR accuracy comparison — the central STRdust
result. No numbers are reported in the paper (
no_expected_result): there is nothing to grade against. The full pipeline (simulate reads → map → phase → call with 3 tools → compare) could be run, but with no reported value it would be a new analysis, not a reproduction. Not attempted as a graded claim. - SURVIVOR read simulation / SNV injection — separate tool; paper gives only one parameter (SNV rate 0.001). Out of scope for the SimiSTR repro.
- All other hackathon sub-projects (Pangenome, etc.) — different code/data, not the harvested RU.
- Wet-lab / manual / external content — none here (all computational).
Determinism caveat (important for grading)
SimiSTR calls random.random() and numpy.random.binomial with no seed. Output
is therefore non-deterministic run-to-run. The repo's shipped
output_fastachr22.fa / output_bedfile.bed are one example run, not a
byte-reproducible target. Reproduction is functional/statistical (the tool runs,
produces the described output structure, and changes obey the documented bounds), not
byte-exact. This is recorded honestly — no exact-match grade is claimable.
Data note
PRJEB5235= "84 tomato accession
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.