MZPAQ: a FASTQ data compression tool.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Reported values were directly comparable
- ✓The central claim held under reproduction
- 🟡Could not use the authors’ exact input data
- 🟡A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough to reproduce 1:1. Reproduced the SRR554369 row of Table 5 (compression ratio = original/compressed) on «our HPC» («job», COMPLETED 13:01). MZPAQ was reconstructed EXACTLY per Methods using the paper's own tool versions: MFCompress v1.01 (read ids+sequences as FASTA; built from github.com/cobilab/mfcompress@b256f850 with -fgnu89-inline) + ZPAQ v7.15 (quality stream, -m5); the '+' stream discarded. Baselines gzip/bzip2/xz(LZMA) and Slimfastq 2.04 (bioconda) run on the same data. CENTRAL CLAIM CONFIRMED: MZPAQ gives the highest ratio and the inter-tool ordering matches the paper (gzip 2.81 < bzip2 3.36 < LZMA 4.32 < Slimfastq 5.06 < MZPAQ 6.09). All five absolute ratios are uniformly ~10-14% BELOW the paper. This uniform offset is a DATA-PROVENANCE difference, not a methods error: we used the full public ENA run (768 MB, 3.32M reads, 100 bp, high-entropy quality), whereas the paper lists SRR554369 as 456 MB = 137 B/read, impossible for 100 bp reads -> the paper used the MPEG-G working-group CURATED SUBSET (smaller, more compressible), which is not individually accession-resolvable, so the gap is not closeable. NOT attempted (hard 20%): Leon (5.48), SCALCE (5.97), LFQC (7.02) - tools not packaged / build-fragile; the other 5 datasets (other rooms); lossless round-trip verification. No fabrication signal: values internally consistent with a clean subset-based explanation.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 58assessed: 2026-06-16 ⛓ f5f9169f7d03
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-16
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-16no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan selecting and combining the best-performing compression sub-algorithms for each individual stream (field) of FASTQ files produce a hybrid tool that achieves the highest compression ratio across all sequencing platforms and data sizes?
- ★ MZPAQ, a hybrid of MFCompress and ZPAQ, achieves the highest compression ratio compared to all evaluated state-of-the-art and general-purpose tools on all benchmark datasets. finding
- ★ MZPAQ offers the best compression ratios regardless of sequencing platform or data size and is compatible with all major sequencing platforms. finding
- ★ By field decoupling FASTQ into separate streams and compressing each with the best-performing algorithm, a superior combined compression tool can be built. method
- ★ MFCompress is the best choice for read identifiers and sequences because it works for all datasets and gives best or second-to-best compression ratios. finding
- ★ LFQC (using the ZPAQ sub-algorithm) gives the best compression ratio for quality scores on all datasets. finding
- Domain-specific compression tools generally outperform general-purpose compressors on genomic data. finding
- MZPAQ discards the third FASTQ stream (the '+' line) during compression and regenerates it at decompression. method
- MZPAQ is provided as a publicly available FASTQ compression tool/resource. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| FASTQ compression benchmarking (compression ratio evaluation) | MPEG HTS benchmark datasets (Illumina and PacBio; H.sapiens, bacteria, yeast, plant, human gut metagenome; 0.5-53 GB) | none | compression ratio (original size / compressed size) | MZPAQ (MFCompress v1.01 + ZPAQ v7.15); compared tools gzip, bzip2, LZMA, Leon, SCALCE, Slimfastq, LFQC, Deliminate, MFCompress |
| Read identifier and sequence stream compression evaluation | 6 MPEG benchmark FASTQ datasets | none | compressed size (MB) and compression ratio for identifier+sequence streams | gzip, bzip2, LZMA, Leon, SCALCE, Slimfastq, LFQC, Deliminate, MFCompress |
| Quality score stream compression evaluation | 6 MPEG benchmark FASTQ datasets | none | compressed size (MB) and compression ratio for quality score stream | gzip, bzip2, LZMA, Leon, SCALCE, Slimfastq, LFQC |
| Decompression fidelity / lossless verification | MPEG benchmark FASTQ datasets | none | ability to reproduce exact replica of original file after decompression | — |
- ▲ MZPAQ achieved the highest overall compression ratio on every benchmark dataset (e.g. SRR327342 8.49, MH0001 7.98, SRR554369 7.04) up to 8.49:1 (SRR327342)
- ▲ On ERR174310 (largest, 53 GB H.sapiens dataset) MZPAQ reached 5.00 vs LFQC which failed (N/A) 5.00:1
- ▲ LFQC gives the best quality-score compression ratio on all datasets (e.g. SRR554369 3.52, MH0001 3.68) 2.59-3.68:1
- ▲ MFCompress had best identifier+sequence ratio on the first dataset and second-to-best on all others e.g. 16.30:1 (SRR554369)
- – Identifier+sequence stream compression ratios ranged from about 4:1 to 16:1 4:1 to 16:1
- – Quality score stream compression ratios ranged from about 2:1 to 4:1 2:1 to 4:1
- – SCALCE and Slimfastq failed on the PacBio dataset (SRR1284073) and LFQC failed on two larger datasets
- ▲ All domain-specific tools outperformed general-purpose tools except LZMA, which failed on PacBio data
- fold_change 8.49 (MZPAQ compression ratio on SRR327342 (highest))
- fold_change 7.04 (MZPAQ compression ratio on SRR554369 vs LFQC 7.02)
- fold_change 16.30 (MFCompress identifier+sequence ratio on SRR554369 (best))
- fold_change 3.52 (LFQC best quality-score ratio on SRR554369)
- count 85 GB publicly available subset of 2.4 TB MPEG benchmark (benchmark dataset size; 6 datasets used)
- other 0.5 GB - 53 GB (range of benchmark dataset sizes)
- fold_change 5.00 (MZPAQ ratio on ERR174310 (53 GB H.sapiens), highest among tools)
- fold_change 3.22 (MZPAQ and LFQC tie on PacBio dataset SRR1284073)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a software engineering benchmarking paper, not a hypothesis-testing study; no inferential statistics are employed. The authors empirically compared compression ratios, speeds, and memory usage of multiple algorithms across six publicly available genomic benchmark datasets by direct tabulation of raw output metrics. Algorithm selection for the composite tool (MZPAQ) was made by inspection: the algorithm achieving the best or second-best compression ratio across the most datasets was selected for each FASTQ sub-stream. Results are reported as ratio values and file sizes in tables, with best and second-best entries highlighted typographically.
-
Each algorithm was run once per dataset and the single resulting file size was used as the performance measure, with no repeated trials↳ Could also: Running each algorithm multiple times under controlled system conditions and reporting mean ± SD (or median ± IQR) of compression time and ratio across runs — Repeated measurement would quantify run-to-run variability attributable to system load, I/O scheduling, and cache state, making speed comparisons more reproducible and allowing formal statistical comparison (e.g., pairwise Wilcoxon signed-rank tests across datasets)
-
Algorithm selection was made by visual inspection of tables, identifying which tool achieved the best or second-best ratio on the most datasets↳ Could also: Ranking algorithms per dataset and summarising with mean rank or a Friedman test followed by post-hoc pairwise comparisons (e.g., Nemenyi test) across the six datasets — A rank-based nonparametric framework would provide a principled, reproducible selection criterion and a p-value or confidence interval for the claim that one algorithm dominates others across datasets, without assuming normality of ratio distributions
-
Compression ratio (original size / compressed size) is reported as the sole summary of compression effectiveness↳ Could also: Also reporting bits per base (bpb) or bits per quality-score symbol as additive, stream-decomposable metrics — Bits-per-symbol metrics are additive across sub-streams and independent of original file size, making cross-dataset and cross-stream comparisons more directly interpretable; they are a standard unit in the information-theoretic literature on sequence compression
-
The six benchmark datasets differ simultaneously in organism, sequencing platform, coverage depth, and file size, and these factors are not separated in the analysis↳ Could also: A factorial or mixed-effects model with algorithm, platform (Illumina vs. PacBio), and genome size class as factors, using compression ratio as the response variable — Decomposing variance by covariate would allow the authors to characterise whether algorithmic differences in ratio are consistent across platforms or interact with data characteristics, supporting more targeted tool recommendations
-
N/A entries (tool failures or unsupported datasets) are excluded from per-cell comparisons without formal handling↳ Could also: Treating failures as censored observations or reporting a structured availability matrix separately from the performance matrix — Explicit missingness handling prevents implicit selection bias in ratio averages and makes coverage guarantees (a stated design goal) a first-class, formally evaluated criterion alongside compression magnitude
-
The paper selects the best sub-algorithm for each FASTQ stream independently (MFCompress for identifiers/sequences, ZPAQ for quality scores) and combines them without evaluating the combined tool on a held-out dataset↳ Could also: Cross-validation or leave-one-dataset-out evaluation of the composite tool to assess whether its superiority generalises beyond the datasets used for sub-algorithm selection — Because MZPAQ's components were chosen by optimising on the same benchmark used to evaluate the final tool, a held-out or cross-validated assessment would provide an estimate of how well the selection generalises to unseen data types
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-31171931 (MZPAQ: a FASTQ data compression tool)
El Allali A, Arshad M. Source Code for Biology and Medicine 2019;14:3. DOI 10.1186/s13029-019-0073-5 · PMCID PMC6547476.
What the paper reports (computational)
A new lossless FASTQ compressor MZPAQ = hybrid of MFCompress v1.01 (read identifiers + sequence streams, treated as FASTA) and ZPAQ v7.15 (quality stream). The "+"/separator stream is discarded and regenerated on decompression. It is benchmarked against general-purpose (gzip, bzip2, LZMA) and FASTQ/FASTA compressors (Leon, SCALCE, Slimfastq, LFQC, Deliminate, MFCompress) on six MPEG-HTS datasets. The headline result is the compression ratio = original_size / compressed_size per tool per dataset (Table 5).
In scope (pipeline-derived, attempted here)
This RU's assigned dataset is SRR554369 (P. aeruginosa, Illumina GAIIx, paired-end). We reproduce the SRR554369 row of Table 5 — the compression ratio of each tool on this dataset. Tools attempted (sourced, deterministic, runnable):
- gzip, bzip2, LZMA (xz) — general purpose, deterministic. CLEAREST points.
- Slimfastq 2.04 (bioconda) — FASTQ-specific.
- MZPAQ — reconstructed exactly as described: MFCompress (built from github.com/cobilab/mfcompress) on a FASTA of (id,seq) + ZPAQ v7.15 (conda-forge, exact paper version) on the quality stream; ratio = orig/(mfc+zpaq). HEADLINE.
Out of scope / not attempted (the hard 20%)
- LFQC — Ruby/Perl wrapper around multiple coders; not packaged, build-fragile;
not attempted (would be
env_unresolvable-ish, not chased). - SCALCE — not in bioconda; needs from-source build with old boost/zlib; skipped.
- Leon — GATB-based; not in bioconda under that name; skipped.
- Deliminate, MFCompress-as-FASTQ-baseline — secondary FASTA baselines; skipped.
- The other five datasets (SRR327342, MH0001, SRR1284073, SRR870667, ERR174310) — out of scope for this RU (this room reproduces SRR554369 only).
- Decompression / lossless round-trip verification — not attempted (compression ratio is the reported claim).
Data
- Source: ENA HTTPS mirror (same SRA run as the benchmark's DDBJ bz2 files): ftp.sra.ebi.ac.uk/vol1/fastq/SRR554/SRR554369/SRR554369_{1,2}.fastq.gz
- Benchmark (sfu-compbio/compression-benchmark, samples.md) uses the paired files SRR554369_1 + SRR554369_2 (DDBJ), ~1 GB uncompressed for the pair. The MZPAQ paper lists SRR554369 as "456 MB"; we process the concatenated pair and report the size we actually used (the ratio, not the byte count, is the compared claim).
Compute
All compression runs on «our HPC» compute node (SLURM job, std partition). Data + env
- tools live on «infra». «host» holds only small result values + pointers.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
The method was reproduced 1:1 using the paper's own tool versions (MFCompress 1.01 + ZPAQ 7.15), and the central claim holds: MZPAQ achieves the highest compression ratio with the reported inter-tool ordering preserved. However, all five absolute ratios are uniformly ~10–14% below Table 5 (MZPAQ 7.04→6.09), traceable to a data-provenance problem on the authors' side: the paper used a non-deposited MPEG-G curated subset, and its stated size (456 MB = 137 B/read for 100 bp reads) is physically impossible, so the absolute values are not derivable from the public accession. No fabrication signal — the offset is internally consistent with a smaller, more-compressible subset — but the headline numbers are not independently verifiable and 3 of 8 tools were not attempted.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.