KAGE: fast alignment-free graph-based genotyping of SNPs and short indels.
Provisional — an automated or curator check raised a specific concern and points reviewers here. This is NOT a final assessment and not a determination about the authors.
The main result did not reproduce in this reproduction attempt. Where our recomputation produced values that differ from the published ones, those discrepancies are listed below. This is a single automated attempt — not peer review and not a finding of error or misconduct — and differences can also arise from data access, undocumented parameters or the computing environment. The verdict can be contested via “report an error”.
- ✓No authors-side cause for any deviation
- 🔴Could not use the authors’ exact input data
- 🔴Reported values were only indirectly comparable
- 🔴A deviation arose in the data or preprocessing
- 🔴Reported values were not (fully) derivable from the shared data
- 🔴The deviation was non-trivial in magnitude
- 🔴The central claim did not (fully) hold under reproduction
- 🔴Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
REPRODUCED (partial, healthy). Ran the authors' OWN benchmark pipeline (github.com/ivargr/genotyping-benchmarking @ v0.0.1, commit 5400d1b) END-TO-END on «our HPC» SLURM: the README <1hr simulated quick experiment. Simulated a 1000-haplotype population over 10 Mbp of chr1, simulated 15x reads, ran KAGE (kmer_mapper + kage genotype, usN1000) and evaluated with hap.py. RESULT: KAGE SNP recall/prec/F1 = 0.958/0.977/0.967 (paper Table-3 real-HG002: 0.929/0.981/0.955) - reproduces the SNP-accuracy regime closely (F1 within ~1.3%). INDEL F1 0.977 vs reported 0.709 - higher BY DESIGN (simulated indels are far simpler than real GIAB indels), an expected divergence, not a contradiction; no fabrication signal. The EXACT Table-3 numbers need real HG002 reads + the 1000G v2a (AF>0.1%) variant panel, hosted on the authors' now-OFFLINE host ivarg.ddns.net (hg38 substituted from UCSC); the real-data path is blocked by data-host loss. Table 1, Figure 4, and the PanGenie arm not attempted. NOTE: the pipeline did NOT run as-shipped on a modern system - required reconstructing the Dec-2021 pip stack (graph_kmer_index 0.0.14, shared_memory_wrapper 0.0.4, SharedArray ABI source-rebuild, numpy 1.20.3), fixing two missing snakemake conda: directives (repo bugs), a modern bcftools wrapper, GNU time, and an rtg thread-cap (full list in REPRODUCTION_NOTES.md + kartei card kage-genotyping-benchmarking).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-19 ⛓ 201188a0788e
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-29
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-07-31
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetAlignment-free genotyping accuracy can be improved by (1) modeling each kmer's expected count from population data (to handle non-unique kmers) and (2) using a single 'helper' variant to adjust genotype priors, an approach simpler than HMM-based methods (e.g., PanGenie) that scales to thousands of haplotypes without the quadratic cost.
- ★ KAGE combines population-based kmer count modeling with single-variant prior adjustment into an alignment-free genotyper that matches the accuracy of the best existing alignment-free genotypers while being an order of magnitude faster. finding
- ★ Modeling expected kmer counts from population data (vs. naively assuming kmer uniqueness) substantially improves genotyping accuracy, especially for variants with non-unique kmers. finding
- ★ Using a single helper variant to compute genotype priors lets KAGE efficiently leverage information from thousands of haplotypes (5096 from 1000 Genomes), unlike PanGenie's HMM approach whose state space grows quadratically with haplotype count. mechanism
- ★ KAGE achieves accuracy comparable to the alignment-based genotyper Graphtyper and to PanGenie, while being more than 30 times faster than Graphtyper and using far less memory and time than PanGenie. finding
- ★ Full variant calling with GATK achieves higher genotyping accuracy than any of the genotyping-only methods tested, but at much greater computational cost. finding
- ★ Running the imputation tool GLIMPSE on KAGE's output genotype likelihoods further increases accuracy over KAGE's built-in single-variant imputation. finding
- A Snakemake pipeline for reproducing all benchmarking results is publicly released. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| alignment-free kmer-based genotyping (prototype/naive vs. population kmer-count model) | small benchmark dataset, reads simulated (human, HG002-derived) | none (method comparison) | genotyping accuracy vs. Malva | — |
| kmer-based genotyping accuracy stratified by kmer uniqueness | human SNPs (benchmark dataset) | none (method comparison: Naive KAGE vs KAGE+kmer count model vs KAGE full) | accuracy (%) on SNPs with unique vs non-unique kmers | — |
| genotyping with varying numbers of input haplotypes (KAGE vs PanGenie) | small test dataset, 5 Mbp of chromosome 1, reads simulated from HG002 | number of individuals/haplotypes included in model | recall/precision (F1 score) of predicted genotypes | — |
| genome-wide short-read genotyping benchmark (KAGE, KAGE+GLIMPSE, PanGenie, Bayestyper, Malva, Graphtyper, GATK) | human individual HG002 (GIAB), 1000 Genomes Project variant set | none (tool comparison), short reads at 15x coverage | recall, precision, F1 (SNPs and indels), runtime, memory usage vs. GIAB truth set | 16 compute cores (all tools except Malva) |
| short-read genotyping validation at alternate coverage | human individual HG002 (GIAB) | read coverage increased to 30x | recall/precision/F1 compared to 15x results | — |
| short-read genotyping validation on independent sample | human individual HG006 (GIAB) | none (held-out sample, not used during method finalization) | recall/precision/F1 compared to HG002 results | — |
| post-hoc statistical imputation on genotype likelihoods | HG002 VCF output from KAGE (without built-in imputation) | GLIMPSE imputation applied | recall/precision/F1 (SNPs and indels) vs. KAGE alone | GLIMPSE |
- – Naive kmer-counting genotyper performs worse than Malva; version modeling expected kmer counts from 1000 Genomes population data performs better than Malva
- ▲ Accuracy on SNPs with non-unique kmers rises from Naive KAGE to KAGE with kmer count model to KAGE (full); unique-kmer SNP accuracy stays high throughout 64% → 77% → 92% (non-unique); 97% → 97% → 99% (unique)
- ▲ Accuracy (F1) of both KAGE and PanGenie increases as the number of individuals/haplotypes included in the model increases
- ▲ On HG002 (15x), KAGE reaches SNP F1 0.955 / indel F1 0.709 in 12 min using 18 GB, versus PanGenie (F1 0.941/0.699, 3.9 h, 111 GB) and Graphtyper (F1 0.949/0.708, 5.1 h, 23 GB) KAGE >30x faster than Graphtyper; PanGenie needs >20x KAGE's runtime
- ▲ GATK achieves the highest accuracy of all methods tested (SNP F1 0.973, indel F1 0.928) but requires 9.0 hours and 64 GB memory 9.0 hours runtime
- ▲ KAGE + GLIMPSE improves accuracy over KAGE alone (SNP F1 0.968 vs 0.955; indel F1 0.713 vs 0.709) at the cost of longer runtime (1.4 hours vs 12 min) SNP F1 +0.013; indel F1 +0.004
- – Results on HG006 and at 30x coverage were similar to the main HG002 15x results
- fold_change KAGE is >30 times faster than Graphtyper (runtime comparison at comparable accuracy, HG002 15x benchmark)
- fold_change PanGenie requires more than 20 times KAGE's runtime (HG002 15x benchmark runtime comparison)
- count 5096 haplotypes used by KAGE from the 1000 Genomes Project (full population reference panel used for KAGE (2548 individuals))
- count 64 input haplotypes (32 individuals) used for PanGenie (PanGenie limited due to quadratic scaling of HMM states)
- other 64% / 77% / 92% (accuracy on non-unique-kmer SNPs for Naive KAGE / KAGE+kmer model / KAGE full (Table 1))
- mean SNP F1 0.955, Indel F1 0.709 (KAGE accuracy on HG002 (Table 3))
- mean SNP F1 0.973, Indel F1 0.928 (GATK accuracy on HG002 (Table 3), highest among compared methods)
- other ~2 minutes genotyping + ~10 minutes kmer counting = ~12 min total; 18 GB memory (KAGE runtime/memory breakdown for HG002 experiment)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a computational genomics methods paper benchmarking a new genotyping algorithm (KAGE) against existing alignment-based and alignment-free genotypers (PanGenie, Malva, Bayestyper, Graphtyper, GATK). Performance was assessed by computing recall, precision, and F1 score for predicted genotypes against truth datasets (GIAB HG002 and HG006), along with runtime and memory usage, rather than through classical inferential hypothesis testing. Results are reported as point estimates on specific benchmark datasets (including one described explicitly as 'a single small benchmark dataset'), without dispersion measures, p-values, or confidence intervals.
-
Genotyping accuracy is summarized as single point estimates (recall, precision, F1) per method and dataset, without any measure of variability.↳ Could also: Bootstrap resampling of variants (or repeated subsampling of reads) to generate confidence intervals around recall/precision/F1 — This would convey how much the observed accuracy differences between genotypers could be attributed to sampling variability versus a stable difference in method performance.
-
Comparisons between KAGE and other genotypers (e.g., PanGenie, Graphtyper) are presented as accuracy differences without formal significance testing.↳ Could also: A paired test appropriate for per-variant correct/incorrect calls across methods, such as McNemar's test or a permutation/bootstrap test on the paired genotype calls — Since all methods are evaluated on the same variant set and individual, a paired approach could formally quantify whether an observed accuracy gap reflects a systematic effect rather than chance variation in a single dataset.
-
The kmer-count modelling experiment (Idea 1, Fig. 3, Table 1) and the helper-variant experiment (Idea 2, Fig. 4) are each described as being run on one benchmark dataset.↳ Could also: Replicating the same comparison across multiple individuals or simulated read sets and reporting the accuracy distribution (e.g., mean ± SD or range across replicates) — Multiple replicate datasets would let readers see how consistent the accuracy improvement from each idea is across different genomic contexts or individuals, complementing the single-dataset demonstration already presented.
-
The truth dataset used for benchmarking (GIAB) was itself partly constructed using GATK, one of the tools being compared.↳ Could also: Using or additionally reporting results against an independently derived truth set not involving any of the benchmarked callers, or stratifying accuracy by how the truth-set regions were validated — An independently constructed reference could help separate genuine differences in genotyping accuracy from any relationship between a benchmarked tool and how the truth set was built, a possibility the authors themselves note as worth considering.
-
Runtime and memory usage are reported as single measurements per tool run.↳ Could also: Reporting these metrics across multiple runs (mean ± SD) or on multiple hardware configurations — Repeated runs would characterize variability in runtime/memory that can arise from system load, I/O, or caching effects, giving a fuller picture of expected performance in other computing environments.
-
Accuracy metrics (recall, precision, F1) are reported without confidence intervals or hypothesis tests when comparing KAGE with and without GLIMPSE imputation.↳ Could also: A paired statistical comparison (e.g., Wilcoxon signed-rank test across variant subsets or genomic bins) between KAGE and KAGE+GLIMPSE — This would let readers assess whether the accuracy gain attributed to adding GLIMPSE imputation is consistent across different regions of the genome rather than being reported as a single aggregate value.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-36195962 (KAGE: alignment-free graph-based genotyping)
Grytten I, Dagestad Rand K, Sandve GK. Genome Biol 2022. PMID 36195962 / PMC9531401 DOI 10.1186/s13059-022-02771-2
What the paper reports (candidate claims)
KAGE is a fast, alignment-free, graph-based genotyper for SNPs/short indels. Two benchmarks dominate the quantitative results:
- Table 1 — accuracy contribution of the two novel ideas (kmer genotype-likelihood model) on non-unique vs unique kmers (Naive→Idea1→Full): 64/77/92% (non-unique), 97/97/99% (unique).
- Table 3 — head-to-head on HG002 @15x (GRCh38, 1000G v2a variants AF>0.1%): SNP & indel recall/precision/F1, runtime, and peak memory for KAGE vs PanGenie, BayesTyper, Malva, Graphtyper, GATK. Headline: KAGE ~12 min vs hours for others; accuracy "at par with best alignment-free genotypers"; >30x faster than Graphtyper.
- Figure 4 — accuracy vs number of population haplotypes (simulated, ~5 Mbp chr1), KAGE vs PanGenie.
- Combined with GLIMPSE imputation, KAGE exceeds any single method (secondary).
In scope (pipeline-derived, reproducible)
All Table-3 / Table-1 / Fig-4 numbers come from the Snakemake benchmark in github.com/ivargr/genotyping-benchmarking (genotyper runs + hap.py evaluation).
- Primary target (designed quick path):
simulated_experiment.smk— simulates reads with graph_read_simulator and runs KAGE (usN1000) vs PanGenie on simulated_dataset2 @15x, evaluates with hap.py → result table. README claims <1 hr. - Stretch target: the HG002 @15x real-data Table-3 row for KAGE (authors' own tool), accuracy + runtime. Full 6-genotyper Table 3 = 2–3 days, 16 cores/method → only if the quick path succeeds and time permits.
- Minimum fallback (P16): install KAGE (
pip install kage-genotyper, authors' own tool) and reproduce its self-test / a small genotyping run end-to-end, confirming the tool runs and emits the expected accuracy on a controlled input.
Out of scope
- GPU variant GKAGE timing (needs GPU; separate paper/section) — attempt only if trivial.
- GLIMPSE-combined results (extra external tool, secondary claim).
- Wet-lab / GIAB truth-set construction (external, not a pipeline we re-run).
Data note
"Data" accession in registry = Zenodo 10.5281/zenodo.7065145, which is ONLY the KAGE source archive (kage-0.1.1.zip, 74 KB) — NOT benchmark data. Actual benchmark inputs are fetched from public sources by the pipeline: UCSC GRCh38 2bit, 1000 Genomes v2a variant VCF, GIAB truth (HG002/3/4/6), Illumina Polaris SV VCF. Profiled accordingly.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.