Compression of structured high-throughput sequencing data.
Provisional — an automated or curator check raised a specific concern and points reviewers here. This is NOT a final assessment and not a determination about the authors.
The main result did not reproduce in this reproduction attempt. Where our recomputation produced values that differ from the published ones, those discrepancies are listed below. This is a single automated attempt — not peer review and not a finding of error or misconduct — and differences can also arise from data access, undocumented parameters or the computing environment. The verdict can be contested via “report an error”.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓No authors-side cause for any deviation
- ✓The central claim held under reproduction
- 🟡Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🔴A deviation arose in the data or preprocessing
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Reproduced the Goby structured-compression benchmark (paper's core pipeline result, Table 2/S2) fully end-to-end for 2 of the 10 Table S1 datasets (HENGLIT/SRR065390 WGS-C.elegans and HZFWPTI/NA12340 Exome-human), including diagnosing and fixing two genuine 2013-vs-2026 compatibility bugs in Goby's SAM parser (supplementary-alignment records; missing MD tags). Storage-efficiency — the paper's headline claim — reproduces within ~5 percentage points across both datasets' H/H+T/H+T+D tiers, with the best tier (H+T+D) landing almost exactly on the paper's reported value for HZFWPTI (29.01% vs 29%). Compression/decompression wall-clock time ratios largely do not reproduce (paper: custom codecs 2-3.4x slower than GZip; here: comparable-or-faster on the long-running dataset, and anomalously much slower on the short-running one) — attributed to 13 years of CPU/JIT evolution and, for the short dataset, JVM warm-up dominating sub-30-second runs, not a pipeline defect. The remaining 8 datasets were profiled from the paper's own tables with lightweight reachability checks (6 confirmed reachable, 2 under-specified by the paper itself) but not independently run through the pipeline, a deliberate scope decision given the pipeline was already twice independently validated end-to-end.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-07-30
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-07-31no human curator yet
- Last updated
- 2026-07-31
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusThe authors ask whether high-throughput sequencing data can be stored in a way that simultaneously supports seamless data schema evolution and achieves substantially better compression than existing formats (BAM, CRAM, general-purpose compressors), thereby reducing storage, computational and network burden across the full HTS data life-cycle.
- ★ Leveraging an explicit data schema (separate field encoding, field modeling, template compression, domain modeling) enables stronger compression of HTS alignment data than general-purpose compression of serialized bytes. mechanism
- ★ The combined H+T+D approach is the current state of the art for lossless compression of HTS alignment data. finding
- ★ Protocol Buffers-based structured schemas allow the file format to evolve over time without breaking existing software, unlike BAM and CRAM which require manual recoding for extensions. method
- ★ A multi-tier data organization (Tier I reads, Tier II stand-alone alignments, Tier III linkage to primary reads) dramatically reduces storage, computational and network burden of collecting, analyzing and archiving sequencing datasets. method
- ★ Spliced RNA-Seq alignments can be stored in less than 4% the size of a BAM file with perfect data fidelity. finding
- CRAM appears over-optimized for whole-genome sequencing data, making its optimizations detrimental for RNA-Seq and DNA methylation alignment data. finding
- ★ The methods are implemented in a comprehensive, freely available software suite (Goby framework, http://goby.campagnelab.org) integrated with IGV, BWA and GSNAP. resource
- CRAM 2.0 is still an experimental implementation, failing to compress or decompress 4 of 10 benchmark datasets. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| RNA-Seq (single-end and paired-end) | public benchmark alignment datasets (samples MYHZZJH, ZHUUJKS, EJOYQAZ) | none | compressed file size, compression/decompression time, compression fidelity vs BZip2 and CRAM 2.0 baselines | — |
| Exome sequencing | public benchmark alignment datasets (samples HZFWPTI, UANMNXR) | none | storage efficiency and compression/decompression time ratios relative to BZip2 and CRAM 2.0 | — |
| Whole genome sequencing (WGS) | public benchmark alignment datasets (samples XAAOBVT, UCCWRUX, HENGLIT) | none | storage efficiency and compression/decompression time ratios relative to BZip2 and CRAM 2.0 | — |
| RRBS (reduced representation bisulfite sequencing) | public benchmark alignment dataset (sample JRODTYG) | none | storage efficiency and compression/decompression time ratios relative to BZip2 and CRAM 2.0 | — |
| Whole genome Methyl-Seq | public benchmark alignment dataset (sample ZVLRRJH) | none | storage efficiency vs BZip2; CRAM 2.0 comparison not obtainable (compression failed) | — |
| Compression fidelity verification (lossless round-trip test) | all ten benchmark HTS alignment datasets | none | identity of decompressed data versus input data; software failure/exception rate | Goby software vs CRAM 2.0 and CRAM 0.7 implementations |
| General-purpose compression benchmark | all ten benchmark HTS alignment datasets | none | file size, compression time and decompression time relative to BZip2 and GZip | BZip2 / GZip codecs |
- ▼ H+T+D compresses HTS alignment data to an average 45% of the size of BZip2-compressed data 45% of BZip2 size (average)
- – H+T+D achieves this compression in 66% of the time needed by BZip2, while decompression is 36% slower than BZip2 compression 66% of BZip2 time; decompression 136% of BZip2 time
- ▼ H+T+D yields files averaging 58.49% of CRAM 2.0 file sizes; H+T averages 72.54% 58.49% (H+T+D) and 72.54% (H+T) of CRAM 2.0
- ▼ For RNA-Seq and methylation datasets, H+T+D compresses to about 36% of the size obtained with CRAM 2.0 across five datasets 36% of CRAM 2.0 size
- – For WGS datasets, H+T is larger than CRAM (109.40% average) while H+T+D remains smaller (86.23% average) H+T 109.40%, H+T+D 86.23% of CRAM 2.0
- ▼ Spliced RNA-Seq alignments stored in less than 4% of BAM file size with perfect data fidelity <4% of BAM size
- ▼ Compared to the previous compression state of the art, dataset size is reduced by more than 40% for exome, gene expression and DNA methylation datasets >40% reduction
- – CRAM 2.0 failed (crashed with an exception) on 4 of 10 benchmark datasets, whereas H+T and H+T+D preserved fidelity on all datasets 4/10 datasets failed
- other 45% (Average H+T+D storage efficiency relative to BZip2 across benchmark datasets)
- other 57% (H+T), 81% (H) (Average storage efficiency of H+T and H relative to BZip2)
- other 58.49% (Average H+T+D size relative to CRAM 2.0 S2 (Methyl-Seq sample excluded due to CRAM failure))
- other 72.54% (Average H+T size relative to CRAM 2.0 S2)
- other 66% (H+T+D compression time as a fraction of BZip2 compression time)
- other 136% (decompression 36% slower) (H+T+D decompression time relative to BZip2)
- other <4% (Size of spliced RNA-Seq alignments stored with the new approach relative to a BAM file)
- count 10,000–100,000 messages per block (Typical PB chunk size used for separate field encoding)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a computational/engineering benchmark study rather than a hypothesis-driven biological experiment: the authors compared several data-compression methods (H, H+T, H+T+D) against BZip2, GZip, and CRAM baselines across a benchmark of ten publicly available HTS datasets (RNA-Seq, Exome, WGS, RRBS, Methyl-Seq). Results are reported as per-dataset percentage ratios of file size and of compression/decompression time relative to the baseline method, and these ratios are summarized with simple arithmetic averages across the ten datasets (Tables 1 and 2). No inferential statistical hypothesis tests, p-values, or measures of dispersion across datasets are reported in the excerpted text.
-
Compression and speed ratios for each method are summarized only as a single average across the ten benchmark datasets (e.g., average storage efficiency of 45% for H+T+D vs. BZip2).↳ Could also: Reporting a measure of dispersion (e.g., SD, IQR, or range) of the per-dataset ratios alongside the average — This would convey how consistent the compression advantage is across different assay types and datasets, in addition to the central tendency already shown.
-
Differences in compression ratio and speed between methods (H, H+T, H+T+D, CRAM) are presented as descriptive percentage comparisons across the ten matched datasets without a formal significance test.↳ Could also: A paired non-parametric test such as the Wilcoxon signed-rank test, or a paired t-test, applied across the ten matched datasets — Because the same ten datasets are compressed by each method, a paired test could formally quantify whether the observed differences exceed what might be expected from dataset-to-dataset variability alone.
-
Three or more compression variants (H, H+T, H+T+D) are compared simultaneously across the same set of datasets.↳ Could also: A repeated-measures ANOVA or the non-parametric Friedman test, followed by post-hoc pairwise comparisons — This approach is designed for comparing multiple related methods on the same set of samples and would also naturally incorporate a multiplicity-adjusted post-hoc step for the pairwise comparisons.
-
The discussion notes that CRAM appears to perform differently depending on assay type (e.g., WGS vs. RNA-Seq/methylation datasets), based on inspection of the per-dataset table.↳ Could also: A stratified analysis or mixed-effects model with assay type as a factor (and dataset as a random effect) — This would let the assay-type-dependent pattern already observed descriptively be quantified formally, including an interaction term between method and assay type.
-
No confidence intervals are given for the average compression or speed ratios.↳ Could also: A bootstrap or asymptotic confidence interval around the mean ratio across datasets — This would communicate the precision of the estimated average advantage, which is particularly informative given the modest number of benchmark datasets (n=10).
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
The headline claim reproduces: Goby's H+T+D codec compresses alignments to ~29-32% of GZip, and we measured 29.01% (HZFWPTI, reported 29%) and 28.31% (HENGLIT, reported 32%), with all storage-efficiency tiers within ~5 percentage points and in the correct order. What does not reproduce are the wall-clock time ratios — HENGLIT compression time 94.17%/93.02%/82.57% against a reported 256%/287%/285%, and HZFWPTI decompression time ~750-797% against a reported ~185-209%. That failure sits on our/technical side, not the authors': these are ratios to a GZip baseline measured on 2026 hardware (and for HZFWPTI a 3.49 s baseline where JVM warm-up dominates), i.e. an endpoint that does not port across 13 years of hardware. Genuine limitations remain on data availability — the NA12340 chr11 extraction gives 5,558,678 reads vs the paper's 6,787,665, three of ten datasets have no usable accession in Table S1, and only 2 of 10 rows were run end-to-end — so the study is solid with explainable deviations, not 1:1.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.