Determining the quality and complexity of next-generation sequencing data without a reference genome.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓No relevant deviation in data/preprocessing
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- ✓Overall, the reproduction was clean
- 🟡Reported values were only indirectly comparable
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
kPAL (k-mer profile Analysis Library) is a third-party-reusable reference-free NGS QC tool; the paper's own worked-example tutorial and its GEUVADIS RNA-Seq case study are both pipeline-derived and in scope. (1) The kPAL 2.1.1 tutorial pipeline (kpal count/merge/matrix on toy data) reproduces the documented distance-matrix output values EXACTLY (to 3 decimals). (2) Using the paper's own precomputed k-mer profiles for 54 GEUVADIS RNA-Seq samples (downloaded fresh from the ArrayExpress E-GEUV-3 FTP mirror, which still holds the full reported N=665), we recomputed all pairwise scaled+smoothed K12 distances with kPAL's own distance engine and reproduced the paper's headline qualitative claim WITHIN-TOL: sample NA18861.4 is quantitatively the most distant sample from the rest (rank 1/54, ~50% higher mean distance than runner-up), matching the paper's 'by far the highest distance' statement, and the other three named QC-failed samples also rank in the top 5 while the two QC-passed benchmark samples rank in the bottom third. (3) We did NOT reproduce the paper's exact correlation coefficient (R=-0.62) between smoothed-K12 distance and percentage of exonic reads, because that requires exon-mapping rates from a separate external GEUVADIS RNA-Seq alignment pipeline not shipped with kPAL or trivially downloadable as a table -- flagged as not attempted, not as a mismatch. (4) We also did not compute the full 665-sample distance matrix (O(n^2) cost) -- only the paper's own 6 named benchmark samples plus 48 controls (54 total). No fabricated results: every reported number here comes from a file on «infra» scratch (matrix_checkpoint.tsv, geuv_analysis_result.json, outlier_check_result.json) that a human reviewer can re-open and check by hand.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-08-07
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-08-07no human curator yet
- Last updated
- 2026-08-07
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan the quality, complexity and comparability of next-generation sequencing datasets be assessed without any reference genome, by analyzing and comparing k-mer frequency profiles? The authors propose that distances between library-size-corrected k-mer profiles reveal technical artefacts and biological complexity that reference-based QC misses.
- ★ kPAL, an open-source alignment-free package, assesses sequencing data quality and complexity using k-mer frequency profiles and pairwise distances between them, without a reference sequence. resource
- ★ Scaling and dynamic smoothing of k-mer profiles correct for differences in library size, making distance measures robust to sequencing depth while retaining the specificity of larger k. method
- ★ kPAL detects technical artefacts including high duplication rates, library chimeras, genomic DNA contamination and differences in library preparation protocols/capture kits. finding
- ★ Artefacts missed by standard QC tools (e.g., FastQC) are exposed by k-mer profiles; all 49 WGS datasets passed FastQC yet split by library preparation protocol in k-mer space. finding
- ★ The modality and complexity of the k-mer spectrum differ by genomic information type: coding/transcriptome sequences are unimodal, whole-genome sequences multimodal; contaminated or poorly captured libraries shift toward the whole-genome profile. mechanism
- k = 12 is an appropriate k size for human data, giving a balanced number of nullomers, unique and frequent k-mers; for mixed bacterial metagenomes separation from shuffled sequences levels off once k exceeds 10. method
- Library chimeras and erroneous bases introduce artificial k-mers that enrich rare features in the k-mer spectrum, corroborated by elevated inter-chromosomal pairs, broadened bimodal insert size and increased soft clipping. mechanism
- kPAL captures the complexity and diversity of microbiomes and can be used to study changes in microbial communities. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| k-mer profile analysis (kPAL: counting, scaling, smoothing, shrinking, balance, shuffle, pairwise distance) | 665 human RNA-Seq samples sequenced in seven different laboratories | none (computational comparison of scaled vs smoothed 9-mer and 12-mer profiles) | median pairwise k-mer distance (K distance) per sample; number of nullomers; k-mer frequency spectrum | kPAL, freely available at https://github.com/LUMC/kPAL |
| RNA-Seq alignment and exon-level expression quantification (reference-based QC comparison) | Human RNA-Seq samples (same 665-sample set, incl. NA18861.4) | none | median pairwise Spearman correlation of exon quantification; percentage of reads mapping to exons | — |
| k-mer profile analysis with hierarchical clustering of pairwise distances (smoothed 12-mers) | 49 whole genome sequencing (WGS) samples from nine individuals | two different library preparation protocols (protocol 1 vs protocol 2) | pairwise k-mer distances / clustering by protocol and individual | — |
| Read alignment and alignment-based QC metrics | 49 WGS samples | library preparation protocol 1 vs 2; three different aligners compared | percentage aligned reads, duplication rate, properly paired reads, pairs mapping to different chromosomes, GC content, insert size distribution, soft-clipped base pairs | Stampy, BWA and Bowtie2 aligners; FastQC QC pipeline |
| k-mer profile analysis with principal component analysis and hierarchical clustering (smoothed 12-mers) | 43 whole exome sequencing (WES) samples | different exome capture kits (Agilent vs Nimblegen); variable capture performance | PCA components of pairwise k-mer distances; on-target read rate; duplication rate; strand (plus/minus) balance of k-mer profile | Agilent and Nimblegen capture kits |
| k-mer profile distance comparison against randomly shuffled sequences (k size optimization) | Three modelled metagenomes of 30 bacterial genomes from Firmicutes and Proteobacteria (100:0, 50:50, 0:100 phylum ratios) | in silico shuffling of sequences preserving overall nucleotide composition (ten randomly permuted sets) | distance between metagenome and permuted-set k-mer profiles as a function of k | — |
| k-mer profiling of reference sequences | Human reference genome, exome and transcriptome reference sequences | none | modality of k-mer spectrum, number of nullomers, unique and frequent k-mer counts across k = 9 to 15 | — |
| k-mer profile analysis of microbial communities | A series of microbiomes | none | k-mer profile-based complexity and diversity of microbial communities | — |
- ▲ Smoothing 12-mer profiles restored concordance between alignment-free k-mer distances and post-alignment exon quantification correlations, versus weak concordance for scaled 12-mers R = −0.62 (smoothed K12) vs R = −0.34 (scaled K12); R = −0.63 previously for K9
- ▼ Smoothed 12-mer distances were far less sensitive to total read count per sample than scaled 9-mer and 12-mer distances, and reduced inter-laboratory variation R = −0.33 vs −0.67 and −0.83
- ▲ Detection of known problematic samples falling outside the 95% prediction bounds improved with smoothed 12-mer distances 11 of 12 (smoothed K12) vs 8 (K9) and 5 (K12)
- ▲ Sample NA18861.4, with significant genomic DNA contamination, had by far the highest distance to other samples and a k-mer frequency distribution mimicking the full human reference genome only 4% of reads mapped to exons
- – WGS samples clustered into two groups by library preparation protocol despite all passing FastQC, with no significant differences in alignment rate, duplication rate or GC content alignment 99.7%; duplication 2.0%
- – Protocol 1 libraries showed fewer properly paired reads, more pairs mapping to different chromosomes, a bimodal/broader insert size distribution and more soft clipping, indicating library chimeras properly paired 86.7% vs 95.8%; different-chromosome pairs 10.6% vs 2.1%; soft clipping up to 80 bp
- – WES samples formed four clusters: PC1 separated samples by on-target read rate, PC2 separated successful datasets by capture kit (Agilent vs Nimblegen), PC3 isolated one failed dataset successful datasets averaged 69.9% on-target reads; failed sub-clusters at 11.3% and 49.8%
- ▲ Failed dataset WE10_F1L3_NIM showed extreme duplication that unbalanced plus/minus strand coverage, leaving the k-mer profile imbalanced (k-mers and their reverse complements at different frequencies) 3.7% on-target reads; 80% duplication rate
- correlation R = −0.80 (Number of 12-nullomers vs total number of reads per RNA-Seq dataset)
- correlation R = −0.62 (Smoothed+scaled K12 median pairwise distance vs median pairwise Spearman correlation of exon quantification)
- correlation R = −0.34 (Scaled K12 distance vs exon quantification correlation)
- correlation R = −0.63 (Previously reported K9 profile distance vs post-alignment exon expression correlation)
- correlation R = −0.33, −0.67 and −0.83 (Sensitivity of median pairwise distances to total read counts (smoothed K12 vs scaled 9-mer and 12-mer profiles))
- other 86.7% and 95.8% properly paired reads; 10.6% and 2.1% pairs mapping to different chromosomes (WGS library preparation protocol 1 vs protocol 2)
- other 69.9% on-target reads on average; 3.7% on-target and 80% duplication for WE10_F1L3_NIM; failed sub-clusters at 11.3% and 49.8% (WES capture performance)
- count 665 RNA-Seq, 49 WGS, 43 WES samples and a series of microbiomes; 30 bacterial genomes in modelled metagenomes; k = 12 chosen for human data (Datasets analyzed and k size selection)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a methods paper introducing kPAL, an alignment-free k-mer profiling tool, validated by comparing k-mer-based distance/similarity measures against post-alignment quality metrics across RNA-Seq, WGS, WES, and microbiome datasets. Validation relies mainly on Spearman correlation coefficients between k-mer distances and exon-quantification correlation measures, polynomial curve fitting with 95% confidence intervals, principal component analysis, and hierarchical clustering, rather than a classical hypothesis-testing framework. Group comparisons (e.g., library preparation protocols, capture kits, laboratories) are described largely through descriptive percentages and visual clustering/distribution plots.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Spearman correlation (R) | median pairwise k-mer distance vs. median pairwise Spearman correlation of exon quantification, across K9/K12 scaled and smoothed profiles | — | not stated |
| Spearman correlation (R) | total read counts per RNA-Seq sample vs. number of 12-nullomers, and vs. K distance measures | — | not stated |
| Polynomial (poly2) curve fit with 95% confidence intervals | total reads per sample vs. K distance (9-mer and 12-mer, scaled/smoothed), Figure 2G-I | — | not stated |
| Principal component analysis (PCA) | pairwise distances from smoothed 12-mer comparisons of WES samples, clustering by on-target rate and capture kit | 43 WES samples | not stated |
| Hierarchical clustering | pairwise k-mer distance measures across WGS and WES samples | 49 WGS samples; 43 WES samples | not stated |
-
Correlation strength between k-mer distance and exon-quantification measures is reported as a Spearman R value without an accompanying p-value or confidence interval for the correlation itself.↳ Could also: reporting a p-value and/or a bootstrap or analytic confidence interval alongside each Spearman R — this would convey the statistical uncertainty of the correlation estimate in addition to its point value.
-
Differences between library preparation protocols and other groups (e.g., duplication rate, GC content) are described qualitatively as differing or not differing significantly, without naming a specific statistical test.↳ Could also: a formal two-sample test such as a t-test, Mann-Whitney U test, or permutation test — this would provide an explicit test statistic and p-value quantifying the between-group comparison alongside the descriptive percentages.
-
Many correlations and pairwise distance comparisons are computed across k sizes, samples, and laboratories.↳ Could also: applying a multiple-testing correction such as Benjamini-Hochberg FDR when summarizing many comparisons together — this can help control the false discovery rate or family-wise error rate when numerous correlation or distance comparisons are examined jointly.
-
Differences among more than two groups (e.g., multiple laboratories or capture kits) are shown via distribution plots and clustering rather than a stated formal group-comparison test.↳ Could also: an ANOVA or Kruskal-Wallis test with post-hoc pairwise comparisons across labs or kits — this would give a formal statistical basis for differences observed visually among several groups.
-
Median pairwise k-mer distances are used as the main summary statistic for sample comparisons, with their spread not quantified numerically in the text.↳ Could also: reporting IQR, SD, or a bootstrap confidence interval alongside the median distance values — this would communicate the variability of the distance metric in addition to its central tendency.
-
Sample sizes for each dataset type are reported as fixed counts without a stated power or sample-size justification.↳ Could also: including a power analysis or effect-size-based sample-size rationale — this can clarify the extent to which the chosen sample sizes are expected to detect meaningful differences between groups.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
This is close to a model reproduction: the authors' own tool (kPAL 2.1.1) and their own deposited artefacts (shipped tutorial data; 665 precomputed k=12 profiles at E-GEUV-3, count confirmed exactly) were used, and both attempted claims held — the tutorial distance matrix matched to every reported decimal (0.4152893746 vs 0.415, etc.), and NA18861.4 re-emerged as the extreme outlier (rank 1/54, mean scaled+smoothed K12 distance 0.0401, ~50% above the runner-up), with the other three QC-failed samples at ranks 2/3/5 and the two QC-passed samples at 38 and 51. Nothing deviates on the authors' side; no value looks fabricated or 'too perfect', and the deposited data fully supports the headline conclusion. The two shortfalls are entirely on our side and are scope/compute limits, not disagreements: the R=-0.62 exon-mapping correlation was not attempted because exon-mapping rates come from an external GEUVADIS alignment pipeline, and only 54 of 665 samples entered the O(n^2) matrix (~53 s/pair). Minor friction worth recording for the taxonomy: the deposited profiles use the legacy pre-1.0 kPAL format that today's CLI refuses to read, so a manual conversion step was required before any number could be recomputed.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.