Hobbes: optimized gram-based methods for efficient read alignment.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No relevant deviation in data/preprocessing
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓Overall, the reproduction was clean
- 🟡A deviation was attributed to the published material
- 🟡The central claim did not (fully) hold under reproduction
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Reproduced the hardware-independent pipeline outputs of Table 7 (RNA-seq, GSE20846): Bowtie 1.3.1 Hamming-3 all-mappings paired-end gives 39.93%/18.409M (paper 40.28%/18.399M) and 42.61%/18.893M (paper 42.97%/18.883M) — #mappings match to 3 sig figs, %mapped within ~0.4pp. mm9 knownGene reproduces EXACTLY (55,419 transcripts). Found a paper accession typo: cited SRR047951/047953 are wrong (unrelated PRJNA48333); correct GSE20846 runs are SRR037951/037953, whose N (21.95M/20.16M spots) matches the paper's 21.8M/20M. Timing columns are 2012-hardware-bound (not 1:1 reproducible). Hobbes 1.3 built+indexed OK but its paired all-mappings run did NOT complete (8h and 40min timeouts; then stuck in queue), so Hobbes's own Table 7 row was not confirmed. Nothing fabricated.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-07-28
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-07-31no human curator yet
- Last updated
- 2026-07-31
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusThe paper tests whether gram-based read alignment can be made substantially faster than state-of-the-art mappers by addressing two performance bottlenecks — CPU-intensive gram counting and cache misses during candidate verification — via an optimized q-gram selection procedure for reads and a cache-efficient bitvector filter.
- ★ Hobbes, a gram-based short-read mapper supporting Hamming and edit distance, is faster than all other read-mapping programs tested while maintaining high mapping quality. resource
- ★ A dynamic programming algorithm can select an optimal prefix set of d+1 non-overlapping q-grams from a read that minimizes both the number of inverted lists probed and their total number of elements. method
- ★ Augmenting inverted-list elements with bitvector encodings of the q-gram's neighboring reference characters allows pruning false-positive candidate mappings without accessing the reference sequence, avoiding cache misses. method
- ★ Lemma 1 (Position-Based Prefix): any sequence within Hamming distance d of s must share a gram with every subset P of d+1 non-overlapping q-grams of G(s), because d substitutions can affect at most d non-overlapping grams. mechanism
- Lemma 2 (Generalized Position-Based Prefix): a sequence r within Hamming distance d of s must share at least k grams with every subset of d+k non-overlapping q-grams of s. mechanism
- ★ A 1-bit-per-character encoding with A,T => 0 and C,G => 1 preserves filtering power because the most frequent substitutions (A->G and T->C) map to different bit values. finding
- Hobbes outperforms both existing gram-based and BWT-based methods; BWT-based packages degrade as read length and mismatch count increase and are not designed for finding many mappings per read. finding
- Hobbes supports SAM output and is publicly available at http://hobbes.ics.uci.edu. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Read-mapping benchmark (runtime and mapping quality comparison) | Human genome reference; real and simulated short-read data sets | none (varied read length 35-100 bp, varied error rate, Hamming vs edit distance, all-mappings vs few-mappings mode) | Mapping speed (runtime) and mapping quality/accuracy versus Bowtie, BWA, mrsFAST and RazerS | — |
| Character-substitution frequency profiling during candidate verification | hg18 human reference genome with 2 million 35 bp reads | none | Number of each read-character to reference-character substitution observed while verifying candidate mapping positions obtained from optimal gram prefixes | — |
| Algorithmic complexity/cost analysis of the optimal q-gram prefix dynamic program | In silico: example read GGTCTCACCCTGAACTAA (length 18), q = 5, d = 2 | varying d and q parameters | Number of DP matrix cells populated and asymptotic complexity O(|s|d) | — |
| q-gram inverted index construction and traversal (worked example) | Reference sequence excerpt with 5-gram inverted index | none | Normalized inverted-list positions and per-position gram counts relative to the count-filter threshold T | — |
- ▲ Hobbes is faster than all other read-mapping programs tested (Bowtie, BWA, mrsFAST, RazerS) while maintaining high mapping quality.
- ▲ Hobbes is faster than Bowtie when finding all mapping locations of a read in the human genome within a given Hamming or edit distance. about 5-fold
- ▲ Hobbes is faster than BWA, depending on read length and error rate, when finding all mapping locations within a given Hamming or edit distance. 2-10-fold
- – Hobbes is 2-10 times faster than state-of-the-art packages when finding all mappings per read, and performs comparably when looking for a few mappings. 2-10-fold
- ▲ A -> G and T -> C are the most frequent character substitutions between reads and hg18, motivating the A,T => 0 / C,G => 1 bitvector encoding. 1 382 950 075 (A->G) and 1 232 657 183 (T->C) substitutions
- ▼ The optimal-prefix dynamic program has complexity O(|s|d), and its actual cost decreases as d and q increase because fewer sets of non-overlapping grams exist. O(|s|d); 12 matrix cells for an 18 bp read with q=5, d=2
- – In the worked index example (read length 17, q=5, d=2, T=3), candidate positions 92 and 118 are pruned by the count filter and only position 105 (count 3) survives as a candidate. T = 17 - 5 + 1 - 2*5 = 3
- ▼ The bitvector filter requires only a handful of CPU instructions per inverted-list element (bitwise-AND, bitwise-XOR, popcount, and a threshold comparison) and needs no reference-sequence access.
- count 1 382 950 075 (A (read) -> G (hg18) substitutions, the most frequent class, from 2 million 35 bp reads)
- count 1 232 657 183 (T (read) -> C (hg18) substitutions, second most frequent class)
- count 687 276 051 (A -> T substitutions on hg18)
- count 559 937 841 (A -> C substitutions on hg18)
- count 395 839 922 (T -> G substitutions on hg18)
- count 393 616 199 (G -> C substitutions on hg18)
- fold_change about five times faster than Bowtie; 2-10 times faster than BWA (Speedup for finding all mappings in the human genome within a Hamming or edit distance)
- other 3 x 10^18 candidate locations (Cost of mapping a billion reads against the ~3 billion bp human genome)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This paper presents Hobbes, a gram-based short-read alignment algorithm, and evaluates it by benchmarking runtime and mapping quality against other alignment programs (Bowtie, BWA, mrsFast, RazerS) on real and simulated sequencing data with read lengths from 35 to 100 bp. The evaluation is primarily a computational/algorithmic performance comparison (speed and accuracy of read mapping) rather than an inferential statistical analysis of biological samples; the excerpt provided does not describe hypothesis tests, p-values, or replicate-based statistical comparisons.
-
Performance differences between Hobbes and other read-mapping programs are reported as single runtime/accuracy figures (e.g. speed-up factors) without repeated-run variability.↳ Could also: Reporting runtime across multiple independent runs (e.g. mean and SD or range) or using a formal comparison such as a paired t-test or Wilcoxon signed-rank test on matched runtime measurements — This would convey how consistent the observed speed/accuracy differences are across runs or data sets, which is useful when benchmarking results are later relied upon in downstream tool-selection decisions.
-
Character substitution frequencies used to justify the bit-encoding scheme (Table 1) are presented as raw counts from 2 million reads without an associated measure of uncertainty.↳ Could also: Reporting these frequencies as proportions with confidence intervals, or testing whether substitution rates differ from a uniform/null expectation (e.g. chi-squared goodness-of-fit test) — This would formally quantify how much more likely certain substitutions (e.g. A→G, T→C) are relative to others, supporting the rationale for the chosen encoding.
-
Mapping quality and speed are compared across several programs and read-length/error conditions using descriptive performance summaries.↳ Could also: A design with multiple replicate simulated data sets per condition, summarized with an ANOVA or mixed-effects model treating program and condition as factors — This could help separate the contribution of read length, error rate, and program identity to observed performance differences, and quantify uncertainty in the estimated speed-up factors.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
What deviates: essentially nothing on the checkable side — mm9 knownGene reproduces exactly (55,419 transcripts) and the Table 7 Bowtie column reproduces to 3 significant figures (18.399M→18.409M, 18.883M→18.893M), with %mapped off by only 0.35–0.36pp, attributable to Bowtie 1.3.1 vs the 2011 0.12.x build. Whose side: the one real authors'-side defect is a data-integrity issue rather than a numeric one — the cited runs SRR047951/047953 are digit-typos for SRR037951/037953 and point to an unrelated 12K-read study; the one gap in our reproduction (the Hobbes own-tool row, 40.29%/19.157M and 42.99%/19.393M) is squarely our side, caused by self-imposed 8h/40min wall-time caps since removed, and correctly reported as inconclusive rather than as a failure. Severity: negligible for everything graded; q7 is only limited because the paper's central claim is about Hobbes's own speed and sensitivity, which is both hardware-bound (2012 Xeon) and still unverified pending the re-run. Overall: an exemplary, honest reproduction — green on derivability with no fabrication signal at all.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.