A workflow reproducibility scale for automatic validation of biological interpretation results.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No relevant deviation in data/preprocessing
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- ✓Overall, the reproduction was clean
- Every checked point held up.
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
CLEAN 1:1 REPRODUCTION. Tonkaz reproducibility-scale paper (Suetake et al., GigaScience 2023, giad031). The paper's only pipeline-derived quantitative result is Table 5 - Tonkaz's per-output-file reproducibility-Level (3/2/1/0) counts for 9 comparison cases of deposited workflow run-result RO-Crates. Ran tonkaz v0.2.5 (paper-contemporary release; commit 570cebd, 2023-06-06) from source under Deno 1.25.1 on «our HPC» compute node n109 (SLURM «job»), via the authors' own deno-test driver, on the deposited crates (tests/example_crate/*.json, also archived open/CC0 on Zenodo 7660388). All 9 comparisons (C1-C9) reproduced Table 5 EXACTLY (0/5/0/0; 0/0/0/5; 0/4/0/0; 0/0/0/4; 20/5/0/0; 0/0/0/25; 13/12/0/0; 8/5/7/5; 557/306/1/8); the regenerated logs are identical in level counts to the committed reference logs the table was made from. Described well enough to reproduce; SAME tool on SAME data => exact match, no fabrication signal. NOT attempted (out of scope): re-running the three underlying workflows to regenerate the crates (environment-dependent by design, not what Table 5 reports).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-30
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-30no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan the reproducibility of bioinformatics workflow execution results be automatically evaluated on a fine-grained scale—based on biological feature values representing biological interpretation—rather than by a binary check of whether output files are exactly identical?
- ★ A reproducibility scale of workflow execution results (4 levels) can express reproducibility at higher resolution than a binary same/not-same measure by using biological feature values and thresholds. finding
- ★ Comparing output-file checksums alone is insufficient to verify reproducibility because identical biological interpretation can arise from non-identical files (differing versions, timestamps, heuristics, environments). finding
- ★ Sapporo (extended, GA4GH WES-compliant) generates workflow provenance in RO-Crate format that embeds biological feature values (e.g., mapped reads, mapping rate) for verification. method
- ★ Tonkaz, a command-line tool, automatically verifies reproducibility by comparing biological feature values across two RO-Crate provenances and assigning each output file a reproducibility scale level. resource
- ★ File types are identified via extension mapped to EDAM ontology terms, and appropriate tools (SAMtools for SAM/BAM, VCFtools for VCF) extract feature values for comparison. method
- The verification method comprises two steps: extracting biological feature values from output files/logs, then comparing them to expected values within a threshold. method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| bulk RNA-seq workflow (nf-core), checksum comparison of output files | nf-core RNA-seq pipeline execution | re-execution twice in the same environment | output BAM file checksum and file size; mapping rate as biological feature value | Sapporo WES; SAMtools (RRID:SCR_002105) |
| Workflow provenance generation and reproducibility verification | workflows in CWL/WDL/Nextflow/Snakemake executed via Sapporo + Yevis | different teams/environments (reproducibility) or repeated same-environment execution (repeatability) | reproducibility scale level per output file (Fully/Acceptable/Unacceptable/Not Reproduced) | Sapporo (RRID:SCR_023202), Yevis (RRID:SRC_023204), Tonkaz, RO-Crate |
| File-type-based feature extraction (SAM/BAM read stats; VCF variant stats) | workflow output files typed via EDAM ontology (RRID:SCR_006620) | none | number of reads, mapping rate, number of variants, variant frequency, number of lines, file size | SAMtools (RRID:SCR_002105), VCFtools (RRID:SCR_001235) |
- – Executing the nf-core RNA-seq workflow twice in the same environment produced BAM files with different checksums and file sizes differing by several bytes. several bytes
- – Of the 872 files produced by the nf-core RNA-seq workflow, only 25 files are assigned EDAM ontology types and compared by Tonkaz. 25 of 872 files
- – Default reproducibility threshold of 0.05 is used but is configurable (lower for medical or quality-control applications). 0.05
- count 872 files (total files produced by nf-core RNA-seq workflow)
- count 25 files (files assigned EDAM ontology and compared by Tonkaz)
- other 0.05 (default threshold value for acceptable difference in feature values)
- other >70% (researchers who failed to reproduce another scientist's experiments (Baker survey, cited))
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This paper describes a computational methods/tools contribution (a reproducibility scale and a comparison tool called Tonkaz) rather than a hypothesis-testing study. The approach extracts biological feature values (e.g., number of reads, mapping rate, variant frequency) from workflow output files and compares an 'expected' provenance to an 'actual' provenance using file checksums first, then a fixed difference threshold (default 0.05) to classify each output file into one of four ordinal reproducibility levels (Fully Reproduced, Acceptable Differences, Unacceptable Differences, Not Reproduced). Results are reported as this categorical scale per file rather than through classical inferential statistics.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Threshold-based difference comparison (deterministic decision rule, not a classical inferential test) | Comparison of biological feature values between expected and actual workflow output files (e.g., mapping rate from BAM/SAM via SAMtools, variant frequency from VCF via VCFtools) | — | na |
-
The paper uses a single fixed threshold (default 0.05) to decide whether a difference in biological feature values is 'acceptable.'↳ Could also: Statistical equivalence testing (e.g., two one-sided tests/TOST) against a pre-specified equivalence margin — This would frame the acceptable-vs-unacceptable decision within a formal statistical inference structure with an associated error rate, complementing the current deterministic cutoff.
-
Reproducibility outcomes are summarized only as a categorical scale (0-3) per output file, without a numeric distribution of differences.↳ Could also: Reporting summary statistics (e.g., mean/median relative difference and a dispersion measure such as SD, IQR, or a confidence interval) across the compared files — This would let readers see how far typical differences sit from the threshold, in addition to the pass/fail-style classification.
-
A single default threshold (0.05) is applied uniformly across different biological feature types (e.g., mapping rate, variant frequency, read counts).↳ Could also: Feature-specific or empirically derived thresholds, for example based on the observed variability from repeated executions of the same workflow under identical conditions — This could tailor the acceptable-difference bound to the natural run-to-run variability of each feature type rather than applying one uniform cutoff to all of them.
-
The demonstration compares provenance from workflow executions without reporting how many repeated runs were performed to characterize baseline variability.↳ Could also: Repeated (technical) re-executions of the same workflow in the same environment to build an empirical repeatability distribution — This would help distinguish expected stochastic variation in tool output from meaningful divergence, informing where a threshold might be set.
-
Reproducibility is reported at the level of a categorical scale (Fully Reproduced / Acceptable / Unacceptable / Not Reproduced).↳ Could also: A continuous relative-difference or percent-error metric reported alongside the categorical scale — This would retain more granular information about the magnitude of differences than the four discrete levels alone.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-37150537 (Tonkaz / workflow reproducibility scale)
Paper: Suetake H, Fukusato T, Igarashi T, Ohta T. "A workflow reproducibility scale for automatic validation of biological interpretation results." GigaScience 2023; PMID 37150537; PMCID PMC10164546; DOI 10.1093/gigascience/giad031.
What the paper is
A methods paper proposing Tonkaz, a CLI tool that assigns a reproducibility scale (per output file, Levels 3/2/1/0) to a pair of workflow execution results. It compares an "expected" run-result RO-Crate against an "actual" run-result RO-Crate:
- Level 3 (Fully Reproduced): output files identical (same checksum).
- Level 2 (Acceptable Differences): files differ but biological feature values (read count, mapping rate, variant frequency, …) within threshold (default 0.05).
- Level 1 (Unacceptable Differences): files differ and features beyond threshold.
- Level 0 (Not Reproduced): file missing in the actual execution.
Real tool repo: https://github.com/sapporo-wes/tonkaz (Deno/TypeScript, Apache-2.0).
(The scaffold's code_url github.com/ResearchObject/ro-terms is a mis-resolved
link — ro-terms is the RO-Crate vocabulary, not this tool.)
Pipeline-derived results IN SCOPE
The paper's quantitative computational result is Table 5 — Tonkaz applied to 9 pairs of workflow run-result RO-Crates, reporting the count of output files at each reproducibility level. These counts are the output of the Tonkaz pipeline and are exactly reproducible by re-running Tonkaz on the same deposited crates.
| ID | Source | Target | Use case | L3 | L2 | L1 | L0 |
|---|---|---|---|---|---|---|---|
| C1 | GATK_1st | GATK_2nd | same env | 0 | 5 | 0 | 0 |
| C2 | GATK_1st | GATK_mac | diff env | 0 | 0 | 0 | 5 |
| C3 | JGA_1st | JGA_2nd | same env | 0 | 4 | 0 | 0 |
| C4 | JGA_1st | JGA_mac | diff env | 0 | 0 | 0 | 4 |
| C5 | RNA-seq_1st | RNA-seq_2nd | same env | 20 | 5 | 0 | 0 |
| C6 | RNA-seq_1st | RNA-seq_mac | diff env | 0 | 0 | 0 | 25 |
| C7 | RNA-seq_1st | RNA-seq_v3.6 | diff version | 13 | 12 | 0 | 0 |
| C8 | RNA-seq_1st | RNA-seq_small | missing dataset | 8 | 5 | 7 | 5 |
| C9 | RNA-seq_1st | RNA-seq_2nd | all output files | 557 | 306 | 1 | 8 |
These map 1:1 to the bundled tests in tests/*_test.ts with expected dumps under
tests/comparison_results/. The reproduction = run Tonkaz on the deposited crates
and check the per-level counts equal Table 5.
Data
- Zenodo 10.5281/zenodo.7660388 — "Raw data of workflow execution results used
in Tonkaz's experiments" (~24 GB of zip archives = full workflow outputs per
execution). The RO-Crate metadata JSONs that Tonkaz actually consumes (file
checksums + extracted feature values) are also shipped inside the repo at
tests/example_crate/*.json(gatk/jga/rnaseq 1st/2nd/mac/v3.6/small), so the scale computation needs only those small JSONs, not the 24 GB of raw outputs. - Zenodo 10.5281/zenodo.7102664 — sapporo-wes/test-workflow 1.0.2 (workflow defs).
- Zenodo 10.5281/zenodo.7559433 — Tonkaz software release (code DOI).
OUT OF SCOPE (not attempted)
- Re-running the three underlying workflows (GATK WDL, nf-core RNA-seq Nextflow, JGA CWL) to regenerate the run-result crates. That is environment-dependent by design (it is the very thing the paper studies) and is NOT what Table 5 reports; Table 5 reports Tonkaz's verdict on the already-deposited crates. Regenerating crates would not reproduce the reported numbers and is unnecessary for them.
- Wet-lab / manual content: none (pure software paper).
Reproduction method
- On «our HPC» front1, «infra» work dir:
git clonetonkaz; install Deno; check out the version contemporary with the paper (resolve via git tags / Zenodo 7559433). - Run
tonkaz example_crate/<src>.json example_crate/<tgt>.jsonfor each C1–C9 pair (with the "all files" flag for C9); capture the per-level file counts. Also run the bundleddeno test -A ./tests/*_test.ts.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
Clean 1:1 reproduction. This is a software/methods paper whose only pipeline-derived quantitative result is Table 5 (Tonkaz reproducibility-scale L3/L2/L1/L0 counts for 9 RO-Crate comparison cases). Running the authors' own tool (tonkaz v0.2.5, paper-contemporary commit 570cebd) on the open CC0 deposited crates reproduced all 9 rows exactly (0/5/0/0 … 557/306/1/8), with the C9 L0=8 per-direction count correctly understood. No deviation, no fabrication signal — the discrepancy, if any, is none and the data is fully on the authors' side as openly deposited.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.