Sequence-based pangenomic core detection.
The main results reproduced: recomputed values matched the published ones within tolerance.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No relevant deviation in data/preprocessing
- ✓Any deviation was negligible
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
REPRODUCED. Tool = Corer (github.com/gi-bielefeld/corer; the BRIEF's Panseq link is a comparison tool, not authors' code). Built Bifrost from source (commit b7659dd) + authors' CountCoreKmers/Corer on «our HPC». All FOUR prokaryotic core k-mer fractions of Table 1 reproduced 1:1 from NCBI genomes via the authors' exact recipe (Bifrost colored cdBG -k17 -c -> CountCoreKmers at quorum=ceil(0.95n)): B.animalis 0.272 vs 0.27 (exact), Y.pestis 0.879 vs 0.88 (exact), E.faecium 0.086 vs 0.085 (within 0.1pp), L.mono 0.0675 vs 0.067 (exact). Genome counts 18/48/153/263 exact. Eukaryotic A.thaliana (assembly+read graph k-mer counts, k=21) is the heavier second-tier result, attempted next.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 75assessed: 2026-06-19 ⛓ 7c537abbba30
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-25
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper asks whether pangenomic core detection can be performed directly on raw genome sequences (assemblies or reads) via a colored de Bruijn graph, instead of relying on gene annotations and alignment, while still allowing flexible, tunable core definitions.
- ★ Sequence-based pangenomic core detection can be performed directly on unannotated genome sequences using a colored de Bruijn graph, avoiding bias from error-prone gene annotations method
- ★ The proposed algorithm finds all core and bridging k-mers in a C-DBG in O(|V|) time with respect to the number of vertices method
- ★ Corer, an implementation of the method in C++ using Bifrost, is alignment-free, has a small memory footprint, and accepts assembled genomes or sequencing reads as input resource
- ★ Quorum q and variation-tolerance δ parameters allow flexible core definition tolerant of sequencing/assembly quality variation and evolutionary sequence variability method
- The adapted algorithm on a compacted C-DBG runs in O(m + |V|log(|V|)) time method
- ★ The method is applicable to both prokaryotic and eukaryotic pangenomes and can find core features below the gene level finding
- Panseq's core predictions were highly conservative compared to other tools across all tested datasets finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| gene-based pangenomic core detection | prokaryotic pangenome with gene annotations | none | matrix identifying core genes | Panaroo v1.2.8 |
| sequence-based pangenomic core detection | prokaryotic and eukaryotic pangenomes (assembled genomes or sequencing reads) | none | core and bridging k-mers (core genome) | Corer (C++, built on Bifrost colored de Bruijn graph) |
| synteny-based whole-genome alignment core detection | prokaryotic pangenome assemblies | none | core genome sequence fragments | Panseq (GitHub version, Nov 2017); PGV (GitHub version, Sept 2021) |
- – Basic algorithm identifies core and bridging k-mers in linear time relative to the number of graph vertices
- – Adapted algorithm on compacted C-DBG runs in O(m + |V|log(|V|)) time, improved from the naive compacted approach
- ▼ Panseq's core predictions were highly conservative, consisting only of very few sequence fragments containing a few hundred or no genes across all tested datasets
- other O(|V|) (runtime complexity of basic core/bridging k-mer detection algorithm)
- other O(m + |V|log(|V|)) (runtime complexity of adapted algorithm on compacted C-DBG)
- other version 1.2.8 (Panaroo version used for comparison)
- count a few hundred or no genes (size of Panseq's conservative core predictions across datasets)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a computational/algorithmic methods paper (not an experimental biology study) that introduces a graph-based algorithm (Corer) for sequence-based pangenomic core detection using colored de Bruijn graphs. Evaluation is performed by comparing the tool's output (e.g., predicted core genome content) against other existing tools (Panaroo, Panseq, PGV) across genomic datasets, and by varying algorithm parameters (k, quorum q, variation-tolerance δ) to observe effects on the predicted core. The provided text does not describe inferential statistical hypothesis testing, p-values, or classical dispersion/error reporting typical of wet-lab or clinical studies.
-
The paper evaluates its algorithm by directly comparing predicted core-genome outputs (e.g., number/size of core fragments) between Corer and other tools (Panaroo, Panseq, PGV) across datasets, presented descriptively rather than with formal inferential statistics.↳ Could also: A quantitative benchmarking framework could also be used, such as reporting summary statistics (mean/median and spread) of core-size or runtime differences across multiple independent datasets, paired with a non-parametric test (e.g., Wilcoxon signed-rank test) comparing tools on the same datasets. — This would let readers judge whether observed differences between tools are consistent and likely to generalize beyond the specific datasets shown, complementing the descriptive comparison already presented.
-
The effect of varying the variation-tolerance parameter δ (and quorum q, k-mer length k) on the predicted core genome is explored, apparently through direct observation of output changes across parameter values.↳ Could also: A sensitivity analysis reporting the variability (e.g., range or coefficient of variation) of core-genome size across a grid of parameter values, or a correlation/regression analysis relating parameter values to core-genome size, could also be used. — This would provide a compact, reproducible quantitative summary of parameter sensitivity alongside the qualitative description already given.
-
Algorithmic performance is described in terms of asymptotic time complexity (e.g., O(|V|) or O(m + |V|log|V|)) rather than empirical runtime distributions.↳ Could also: Empirical wall-clock runtime and memory usage could also be reported across repeated runs or multiple datasets, summarized with descriptive statistics (mean ± SD) or shown as a distribution (e.g., boxplot across datasets). — This would complement the theoretical complexity analysis with practical performance evidence, which can vary from theoretical bounds due to implementation and hardware factors.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-35663029 (Corer: Sequence-based pangenomic core detection)
- Paper: Schulz T, Wittler R, Stoye J. "Sequence-based pangenomic core detection." iScience 25(6):104413, 2022. PMID 35663029 / PMC9160775 / DOI 10.1016/j.isci.2022.104413.
- Authors' tool: Corer — moved repo:
https://github.com/gi-bielefeld/corer(originallygitlab.ub.uni-bielefeld.de/gi/corer). GPL-3.0.- NOTE: the BRIEF's
Code:field pointed atgithub.com/chadlaing/Panseq. Panseq is not the authors' tool — it is one of the comparison tools in a different (older) line of pangenome work. The actual artifact to run is Corer. (Per brief P16 a third-party tool would be equally valid anyway, but here the authors' own tool exists and is the correct target.)
- NOTE: the BRIEF's
Method (pipeline) in one line
Corer reads a colored compacted de Bruijn graph (cdBG) built with Bifrost from a set of genome FASTAs, and extracts the core = k-mers shared by ≥ a quorum of the input genomes (with a δ "variation tolerance" gap allowance). The headline numbers are the core k-mer fraction = (#core k-mers)/(#total k-mers in the pangenome graph).
In scope (pipeline-derived, will attempt)
The four prokaryotic core k-mer fractions in Table 1, parameters
k=17, q=0.95·n, δ=60 (Results + STAR Methods):
| dataset | n genomes | reported core fraction (Table 1) |
|---|---|---|
| Bifidobacterium animalis | 18 | 0.27 / 27.1% |
| Yersinia pestis | 48 | 0.88 / 87.8% |
| Enterococcus faecium | 153 | 0.085 / 8.5% |
| Listeria monocytogenes | 263 | 0.067 / 6.7% |
Pipeline per dataset:
- Download the exact N genome assemblies (accessions in Table S1 /
mmc1.xlsx, confirmed counts 18/48/153/263) from NCBI on the «our HPC» front node → «infra». Bifrost build -r <genomes> -c -k 17 -t T -o graph→graph.gfa.gz+graph.color.bfg.Corer -i graph.gfa.gz -c graph.color.bfg -q round(0.95*n) -d 60 -t T -o core.- core fraction = (#k-mers in core graph)/(#k-mers in full graph).
Quick MINIMUM (low-hanging): B. animalis (18 genomes, ~1.9 Mbp each) — smallest, fastest; do first. Then Y. pestis (48), E. faecium (153), L. monocytogenes (263) in increasing cost.
In scope but harder (attempt after the floor)
- Eukaryotic A. thaliana (PRJEB2457): read-graph vs assembly-graph core, k=21. Reported: assembly graph 172 M k-mers, read graph 485 M k-mers; core 140 M (assembly) vs 188 M (reads) k-mers; 438,718 vs 440,737 genes; agreement 376,922 genes with SibeliaZ. Heavy (read assembly/graph). Attempt only if prokaryotic floor is solid.
Out of scope (not pipeline-reproducible here / external)
- Runtime & memory comparisons vs Panaroo/SibeliaZ/Panseq (hardware-dependent; not a faithful 1:1 on different hardware — report qualitatively only).
- Gene-level annotations / Drosophila FlyBase comparison (manual/external curation).
- Theoretical results (Definitions/Theorems) — not computational outputs.
Datasets profiled this pass
- NCBI nucleotide/assembly accessions (Table S1): 482 genomes total across 4 species (primary in-scope data; not a single "accession" but a curated list).
- PRJEB2457 (ENA): A. thaliana Illumina GA-II reads (eukaryotic part).
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.