Taxonomic analysis of metagenomic data with kASA.
The main results reproduced: recomputed values matched the published ones within tolerance.
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
REPRODUCED the core methodological claim (kASA Fig 5 robustness curve) 1:1 by running the authors' own kASA_snakemake benchmark (commit f783753, kASA v1.4.9 prebuilt binary) on «our HPC» SLURM «job» (COMPLETED 6:54). 7 positive + 3 negative reference genomes downloaded from NCBI, reads len 100 simulated at mutation 0-20% (deterministic seed 100), build+shrink+identify_multiple+evalJson over ~5.16M reads/level. Result exactly matches the published kASA signature: precision stays ~1.0 across ALL mutation rates (0%:1.0 -> 20%:0.9966), while sensitivity/F1/MCC decay non-linearly and sensitivity stays non-zero at 20% (no Centrifuge-style collapse). Plot at reproduction/outputs/kASA_fig5_repro.png, curve at kASA_curve.tsv. NOT attempted: real-data Table 2 (SRS147126 saliva genera) — needs a huge under-specified RefSeq index (deferred, honestly out of 1:1 reach); cross-tool comparison; runtime/RAM panels (hardware-specific). Both datasets profiled. Exact Fig5 numbers are in suppl file B (human overlay needed for point-by-point).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-24
- Rubric version
- not recorded
- Assessed by
- —
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper tests whether a k-mer based metagenomic taxonomic analysis tool using amino acid-like encoding of k-mers, a range of multiple k values, and external-memory data structures can achieve high sensitivity and precision comparable to or better than existing tools while requiring substantially less primary memory (RAM).
- ★ kASA achieves high sensitivity and precision by using an amino acid-like encoding of k-mers together with a range of multiple k's mechanism
- ★ kASA enables a customizable, low primary memory footprint by using an index that can largely reside on secondary memory (disk/SSD) rather than RAM method
- ★ kASA can outperform other tools in robustness and accuracy in many cases while keeping primary memory consumption highly customizable finding
- ★ Translating triplets of nucleotides into an amino acid-like encoding reduces storage to five bits per triplet instead of six and improves sensitivity because synonymous DNA mutations do not affect matching in at least one of six reading frames, while precision is maintained since nucleotide-level changes typically alter the amino acid sequence in other frames mechanism
- ★ Using a dynamic range of k values (lower to upper bound, max 25) allows kASA to optimize both sensitivity (short k) and precision (long k) simultaneously method
- The index can be shrunk without information loss under certain constraints (content file ≤65535 entries, lower k ≥7, static 64-bit index), or shrunk with some accuracy loss via random per-taxon k-mer deletion method
- kASA supports adding new reference sequences to an existing index without full rebuilding resource
- kASA is free, open-source C++ software available for Linux, Mac and Windows on laptop, desktop and HPCC platforms requiring at least 5GB RAM resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| per-read taxonomic identification benchmark | synthetic sequencing (metagenomic) data | tool comparison (kASA vs. selection of other tools) | accuracy and robustness of per-read taxonomic identification | — |
| taxonomic profiling benchmark | synthetic data from the CAMI Challenge | tool comparison, evaluated with OPAL | accuracy of the computed taxonomic profile | OPAL; snakemake pipeline |
| taxonomic profiling on real data | real NGS metagenomic dataset(s) | none (applied kASA to real data) | comparison of resulting taxonomic profile to Kraken2 and Centrifuge | — |
- ▼ Amino acid-like encoding stores k-mers in five bits per nucleotide triplet instead of six 5 bits vs 6 bits
- ▲ Omitting read-level scoring (when only taxonomic profiling is needed) speeds up processing ~1.3-fold
- ▼ Lossless index shrinking (under stated constraints) halves index size 50% (half the size)
- – For a read of length 100, a Relative Score below 0.4 is considered insignificant 0.4 threshold
- other five bits per nucleotide triplet instead of six (storage efficiency of amino acid-like k-mer encoding)
- fold_change ~1.3 (speedup from omitting read scoring during profiling-only runs)
- other Relative Score < 0.4 = insignificant (significance threshold for a read of length 100)
- count 2^16 − 1 = 65535 (maximum content file entries allowed for lossless index shrinking)
- other minimum 5GB RAM required (system requirement for running kASA)
- other k range from 1 to 25 (max k=25 corresponds to 75nt) (allowed range of k-mer word length)
- other index halved in size (effect of lossless shrink under stated constraints)
- other default lower k = 7 (chosen so the prefix trie (first 6 letters) fully reduces search space)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a bioinformatics tool paper (kASA, a k-mer based metagenomic taxonomic classifier) whose 'results' consist mainly of computational benchmarking against other classification tools (e.g. MegaBLAST, Kraken, Kraken2, Centrifuge, Clark) on synthetic read datasets, a CAMI Challenge profiling comparison evaluated with the OPAL tool, and an application to real data compared against Kraken2 and Centrifuge. Reproducibility is supported by two snakemake pipelines. The provided excerpt describes accuracy/robustness/precision-style benchmarking rather than classical inferential hypothesis testing, and it ends before any quantitative results tables/figures with numeric outcomes are shown.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| OPAL profiling-assessment metrics (e.g. accuracy/precision/sensitivity/robustness style benchmarking, not a classical significance test) | Comparison of taxonomic profiles on CAMI Challenge synthetic data | — | not stated |
-
Tool comparisons are described in terms of benchmark performance metrics (accuracy, sensitivity, precision, robustness) rather than formal statistical hypothesis tests with p-values.↳ Could also: Non-parametric tests such as the Wilcoxon signed-rank test across matched benchmark datasets, or bootstrap resampling to generate confidence intervals on performance metrics — This would convey the uncertainty and variability behind observed performance differences between tools, in addition to the point-estimate metrics described.
-
Multiple tools are compared across multiple synthetic and real datasets and metrics without a stated correction for multiplicity.↳ Could also: A multiple-comparison correction (e.g. Holm-Bonferroni or Benjamini-Hochberg FDR) applied across the set of pairwise tool comparisons — This would help control the family-wise error rate or false discovery rate when many comparisons are summarized together, which is a standard consideration whenever several pairwise contrasts are drawn from the same experiment.
-
Synthetic benchmark datasets (from snakemake pipelines and the CAMI Challenge) appear to be used to generate reads for evaluation.↳ Could also: Generating multiple independent replicate synthetic datasets (varying random seeds) and reporting the mean and spread (e.g. SD or 95% CI) of performance metrics across replicates — This would characterize how much of the observed performance difference is attributable to stochastic variation in the read-simulation process versus genuine tool differences.
-
Real-data taxonomic profiles from kASA are compared descriptively to those from Kraken2 and Centrifuge.↳ Could also: A quantitative agreement measure such as Spearman/Pearson correlation of relative abundances, or a Bland-Altman-style agreement plot — This would give a quantitative summary of how closely the profiles agree or diverge, complementing a qualitative/descriptive comparison of profiles.
-
The effect of shrinking the index (randomly deleting a percentage of k-mers per taxon) on accuracy is described as an experiment in the supplement, based on one benchmark's data.↳ Could also: Repeating the deletion experiment across multiple random draws at each percentage and reporting mean ± SD/CI of the resulting accuracy — This would quantify how stable or variable the accuracy loss is at a given deletion percentage, beyond a single deterministic run.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-33784400 (kASA: Taxonomic analysis of metagenomic data)
Paper: Weging, Gogol-Döring, Grosse. Nucleic Acids Res 2021. DOI 10.1093/nar/gkab200. Tool repo (kASA): https://github.com/SilvioWeging/kASA (commit 51a43f8) — C++ taxonomic classifier. Benchmark repo: https://github.com/SilvioWeging/kASA_snakemake (commit f783753) — Snakemake pipeline that downloads reference genomes, simulates/mutates reads, runs each tool, and computes accuracy.
This is a fully computational paper (a method + benchmark). No wet-lab steps → nothing is out-of-scope on those grounds. The repo is the authors' own code (P16 satisfied directly).
IN SCOPE (pipeline-derived) — what we attempt
| # | Reported result | Paper loc | Pipeline | Status |
|---|---|---|---|---|
| C1 | kASA accuracy vs mutation rate: sensitivity, precision, F1, MCC for simulated reads (len 100) at mutation 0–20%, over 7 reference genomes (content.txt) + 3 negatives | Fig 5 (accuracy panel) + Suppl. | benchmark.smk: kASA build → identify_multiple → scripts/evalJson.py |
ATTEMPTING («job») |
C1 is the core methodological claim: kASA degrades gracefully with mutation, keeping high precision. We run the authors' pipeline verbatim with only kASA enabled (all other tool paths set to ""), producing 21 accuracy points (one per mutation level). This is a clean 1:1 of the kASA curve in Fig 5.
HARDER / EXTENDED (floor is not a ceiling) — attempt if feasible
- C2 — Cross-tool comparison (Fig 5 full): same pipeline, additionally building Kraken, Kraken2, KrakenUniq, Clark, Centrifuge, ganon, MetaCache. Each is a separate C++/tool build; reproduces the relative ranking. Large effort — deferred; would extend coverage, not the kASA claim itself.
- C3 — Real-data analysis, SRS147126 (Table 2 oral-microbiome genera): kASA build over a large
RefSeq reference DB +
identifyon the HMP saliva reads (SRR514322), then genus-level profile. Requires a very large reference index (hundreds of GB) not shipped with the repo → HARD; attempt only if the synthetic benchmark completes with headroom.
OUT OF SCOPE / not 1:1 reproducible
- Runtime & peak-RAM benchmarks (time/memory panels): hardware-specific (their CPU/RAM ≠ «our HPC» node) → not numerically comparable; we record our own timings as context only, not as a match.
- kASA
128-bit,shrink, andalphabetsvariants: optional config switches; left off in the primary run to keep the core 1:1 tractable (can be toggled on for a follow-up).
Datasets the paper relies on (profiled in data/dataset_profile.json)
- Reference genomes (NCBI nuccore, 7 positive + 3 negative accessions in content.txt / content_negative.txt) — downloaded + verified here.
- SRS147126 (SRA, HMP human saliva metagenome WGS) — used for the Table 2 real-data analysis.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.