CRISPRbuilder-TB: "CRISPR-builder for tuberculosis". Exhaustive reconstruction of the CRISPR locus in mycobacterium tuberculosis complex using SRA.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
REPRODUCED (1:1, described well enough). CRISPRbuilder-TB (authors' own tool, github cguyeux/CRISPRbuilder-TB @ac50171) was run end-to-end on the paper-named SRA run SRR6407486 = M. tuberculosis H37Ra, as a «our HPC» SLURM job (2230131) building a conda env (blast 2.17, sra-tools 3.4.1, biopython 1.85, python 3.9) and fetching the run via fastq-dump. ALL in-scope targets reproduced exactly: (R1) the pipeline ran without error and emitted all 6 documented output files with a coherent CRISPR-locus reconstruction (ordered spacer array, full type III-A cas operon, supported spacer junctions); (R2) read length 100 bp, k-mer 80, 1,890,901 spots == ENA read_count, base count == ENA base_count, ~86x coverage; (R3) the reconstructed spoligotype resolves to SIT451 via the authors' own str_to_spol/SIT.xls helper, i.e. the H37Rv reference type with the canonical spacers 33-36 deletion -- an exact biological match for H37Ra (the attenuated H37Rv derivative). Minor transparency note: the tool's internal 68-spacer-frame octal (777777477760771) differs cosmetically from textbook classic-nomenclature octal (777777777760771) because frame positions 20/21 are esp30/esp31 extra spacers; the SIT lookup nonetheless resolves correctly to 451 = H37Rv. Not attempted (out of scope for a single-RU reproduction): Table 1 simulated CRISPRs, S7 Table (434 MTBC accessions), Crass comparison. Grades are provisional; AUDIT.md awaits human sign-off.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-18 ⛓ 1bf3253a9f81
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-24
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper tests whether a new short-read-based pipeline (CRISPRbuilder-TB) can exhaustively and accurately reconstruct the Mycobacterium tuberculosis complex CRISPR-Cas locus—including duplications, spacer/DR variants, and IS6110 insertions—more reliably than existing assembly-based tools or classical spoligotyping, and whether publicly available complete MTC genomes contain reliable CRISPR loci.
- ★ CRISPRbuilder-TB is a new pipeline that reconstructs MTC CRISPR-Cas loci directly from short SRA reads without requiring genome assembly method
- ★ CRISPRbuilder-TB outperforms existing tools (Crass, CRISPR_detector, metaCRISPR) in accuracy, robustness, and interpretability of CRISPR reconstruction finding
- ★ A substantial fraction of publicly available MTC complete genomes contain erroneous or unreliable CRISPR loci finding
- ★ Classical in vitro spoligotyping and existing in silico spoligotyping tools miss CRISPR locus diversity such as duplications, spacer/DR variants, and IS6110 insertion locations finding
- ★ CRISPRbuilder-TB achieved perfect reconstruction (precision=1, recall=1) of simulated CRISPR loci across varying read lengths and sequencing depths finding
- ★ CRISPR loci reconstructed by CRISPRbuilder-TB from WGS reads of 7 reference strains matched their complete genome sequences exactly finding
- CRISPRbuilder-TB tool and scripts are publicly available on GitHub and adaptable to other CRISPR-harboring species resource
- CRISPRbuilder-TB identified new rare spacer and DR variants and compiled a catalogue of remarkable CRISPR-associated sequences in MTC finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| De novo CRISPR locus reconstruction from simulated short reads | Simulated MTC CRISPR profiles (3 constructs) | Simulated IS6110 insertions, deletions, duplications, SNP mutations | Reconstructed spacer/DR order, precision/recall | — |
| BLAST-based read filtering against catalogue of remarkable sequences | WGS short reads (SRA) from MTC strains | none | Reads matching CRISPR-related remarkable sequences | BLAST (e-value threshold 1e-7) |
| K-mer/De Bruijn graph contig assembly (CRISPRbuilder-TB) | WGS SRA reads from 434 MTC strains (L1-L6, animal strains, M. canettii) | none | CRISPR contig reconstruction, spacer/DR variants, evolutionary events | Custom python scripts (CRISPRbuilder-TB) |
| CRISPR locus reconstruction from SRA reads compared to reference complete genome | 7 reference MTC strains (4 L4, 2 M. bovis BCG, 1 L2) | none | Concordance between reconstructed and reference CRISPR locus sequence/structure | — |
| In silico spoligotyping | Same 7 reference MTC strains | none | Spacer presence/absence signature | Spolpred |
| Extraction and quality assessment of CRISPR locus from genome assemblies | 187 additional MTC complete genomes from public databases | none | CRISPR locus reliability/error rate, SNP-based lineage classification | Coll et al. SNP scheme; TBminer |
| Comparative benchmarking of CRISPR reconstruction tools | Same simulated read sets as above | none | Number of contigs, multifurcations, spacers recovered | Crass; CRISPR_detector; metaCRISPR |
- – CRISPRbuilder-TB achieved perfect reconstruction of simulated CRISPR loci regardless of read length or depth precision=1, recall=1 (n=80)
- – More than 1/3 of public MTC complete genomes contain largely erroneous CRISPR loci >1/3
- – 25 of 187 (~14%) complete genomes showed dubious CRISPR quality with multiple spacer/DR variations of widely varying sizes 25/187 (~14%)
- – An additional 86 of 187 genomes exhibited repetitive sequence patterns contradicting their SNP-based lineage classification 86/187
- – CRISPR loci reconstructed from SRA reads of 7 reference strains were identical to their complete genome sequences
- ▼ Crass failed to detect any CRISPR in simulated CRISPR-1 and missed or mistook spacers for flanking regions in the other simulated CRISPRs
- ▼ CRISPR_detector failed to identify CRISPR loci for 125bp and 300bp simulated reads and provided only limited DR consensus information
- – 20 rare spacer variants (max 2 SNPs) and 28 DR variants (max 3 SNPs) were identified across 434 MTC strains 20 spacer variants; 28 DR variants
- other precision=1, recall=1 (Simulated CRISPR reconstruction accuracy across read lengths and depths)
- fold_change more than 1/3 (Fraction of public MTC complete genomes with erroneous CRISPR loci (abstract))
- count 25/187 (~14%) (Complete genomes with dubious CRISPR quality)
- count 86 genomes (Genomes with repetitive patterns contradicting SNP-based classification)
- count n=434 (MTC strains used for CRISPR-Cas locus reconstruction and evolutionary analysis)
- count 20 spacer variants (max 2 SNPs per spacer) (Rare spacer variants identified among 68 MTC spacers)
- count 28 DR variants (max 3 SNPs per DR) (DR variants identified in the catalogue of remarkable sequences)
- other e-value = 1e-7 (BLAST threshold for filtering reads similar to remarkable sequences)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a computational methods paper describing and validating CRISPRbuilder-TB, a pipeline for reconstructing CRISPR-Cas loci in Mycobacterium tuberculosis complex from short-read sequences without genome assembly. The analytical approach is primarily descriptive and comparative: the tool is evaluated against Crass, CRISPR_detector, and metaCRISPR on simulated CRISPRs and seven reference strains using precision and recall, and the reliability of 187 public complete genomes is characterised by frequency counts. No inferential statistics or formal hypothesis tests are reported; performance is described qualitatively and via exact precision/recall metrics.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Precision and recall (information-retrieval metrics, not a significance test) | Reconstruction of all simulated CRISPR loci; precision = 1 and recall = 1 reported across all simulated profiles | 80 DVRs reconstructed across simulated CRISPRs | na |
| Frequency counts and proportions (descriptive only) | Assessment of CRISPR quality in complete genomes from public databases (~14% dubious; ~1/3 stated erroneous in abstract) | 187 complete genomes | na |
-
Tool performance was summarised with precision and recall alone, both reported as 1.0 across all simulated profiles↳ Could also: F1 score, Matthews Correlation Coefficient (MCC), or a full confusion matrix (TP/FP/TN/FN) could also summarise reconstruction accuracy — When precision and recall are both perfect the additional metrics are redundant, but in benchmark settings where they differ, F1 or MCC compactly conveys the trade-off and allows single-number comparison across tools and parameter settings
-
Proportions of low-quality genomes in the public database (~14% dubious; ~1/3 erroneous) were reported as point estimates without uncertainty quantification↳ Could also: Wilson or Clopper-Pearson binomial confidence intervals could also accompany these proportions — Confidence intervals convey the precision of frequency estimates and help readers judge whether the true prevalence is likely to be materially higher or lower than the observed sample proportion
-
Tool comparison on simulated data was based on three primary simulated CRISPRs (plus a secondary set) with qualitative description of outputs at fixed read lengths and depths↳ Could also: A Monte Carlo simulation study with randomised CRISPR architectures and a grid of read-length and sequencing-depth conditions could also be used to benchmark tools across parameter space — Broader simulation coverage would quantify how reconstruction performance varies systematically with duplication rate, IS insertion frequency, and depth, providing more generalisable benchmarks than a small fixed set
-
The BLAST e-value filtering threshold (1e-7) was stated as a fixed parameter but was not accompanied by a sensitivity analysis↳ Could also: A threshold sweep (e.g., 1e-5 through 1e-10) with precision/recall reported at each value could also characterise the effect of this choice on downstream reconstruction quality — Reporting performance across a range of thresholds would clarify how sensitive the pipeline is to this parameter and inform adaptation of the tool to other bacterial species with different CRISPR architectures
-
Validation against known ground-truth CRISPR sequences relied on n=7 reference strains representing a subset of MTC lineage diversity↳ Could also: Leave-one-out or cross-validation on the larger n=434 dataset (where consensus across independent runs can approximate ground truth) could also estimate reconstruction accuracy more broadly — Expanding the validation set or applying cross-validation would support claims about generalisability to lineages not represented among the seven reference genomes, particularly for less-studied lineages such as L5 and animal strains
-
Lineage composition of the complete-genome dataset (n=187) was reported as raw counts per lineage without characterising representativeness relative to known global MTC diversity↳ Could also: A chi-squared goodness-of-fit test or multinomial confidence intervals against expected global lineage frequencies could also characterise how representative the database sample is — Formal characterisation of sampling representativeness would help contextualise which lineages are over- or under-represented in the database-quality findings and clarify the scope to which conclusions generalise
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-33667225 (CRISPRbuilder-TB)
Paper: Guyeux C, Sola C, Noûs C, Refrégier G. CRISPRbuilder-TB: "CRISPR-builder for tuberculosis". Exhaustive reconstruction of the CRISPR locus in mycobacterium tuberculosis complex using SRA. PLoS Comput Biol 2021. PMID 33667225 / PMC7968741 / doi:10.1371/journal.pcbi.1008500.
Code: https://github.com/cguyeux/CRISPRbuilder-TB (HEAD ac50171 2022-12-21).
This is the authors' OWN tool (not third-party), but P16 applies either way:
running the published tool on the paper's named data is a valid reproduction.
Data named by RU: sra:SRR6407486 = M. tuberculosis H37Ra (attenuated
lab strain derived from H37Rv), Illumina WGS, paired-end, ~100 bp reads,
1,890,901 spots (ENA: read_count, base_count 378,180,200), study PRJNA427080.
What the tool does (pipeline, fully computational → IN SCOPE)
fastq-dump --split-files --fastadownloads the SRA run to FASTA.makeblastdbbuilds a nucleotide BLAST DB from ALL reads.blastn(evalue 1e-7) of a catalogue of "remarkable sequences" (data/fastas/crispr_patterns.fasta= direct repeats DR, spacers esp1..esp68, IS6110 borders, cas-gene fragments) against the read DB → reads of interest.- Reads transformed to k-mers (k = ⌊4·readlen/5⌋), De Bruijn-style greedy
extension → contigs; matched substrings replaced by annotations (DR0, esp_N,
IS6110, …). →
.contig. - Spacer-pair / duplication analysis →
.not_consecutive,.reads_with_2_spacers. - Cas-locus reconstruction →
.cas_locus,.cas_reads. - IS6110 detection in cas flanks →
.is_around. - (helper
tools.str_to_spol) maps reconstructed spacers → classic 43-spacer spoligotype + SIT lookup viadata/SIT.xls. This is the "95% automated + human finalization" step the paper describes.
In-scope reproduction targets (this RU)
- R1 (functional): tool runs end-to-end on SRR6407486 → all 6 output files produced without error.
- R2 (read metrics, deterministic): printed read length, nb_reads, coverage (= nb_reads·readlen / |NC_000962.3|, genome 4,411,532 bp). Checkable directly from the data.
- R3 (biological 1:1): reconstructed CRISPR locus / derived spoligotype for
H37Ra matches the published H37Rv-type reference spoligotype — canonical octal
777777777760771, i.e. spacers 33–36 absent, all other classic spacers present. The paper validates the tool against complete reference genomes (S3 Table: "100% concordant"); H37Ra/H37Rv is the type reference, so it is exact ground truth.
Out of scope (not attempted)
- Wet-lab / manual curation steps beyond the automated output.
- Table 1 (three simulated CRISPRs) — needs the authors' read simulator inputs, not derivable from SRR6407486.
- S7 Table (434 reconstructed CRISPRs across MTC) — needs hundreds of other SRA accessions; out of scope for a single-RU reproduction. The pipeline that would produce them is the same tool, demonstrated on SRR6407486.
- Crass comparison (S2 Text) — a different external tool.
Compute plan
Env + read download on «our HPC» front1 (internet). makeblastdb + blastn +
Python reconstruction run as a SLURM job (--partition=std) reading prepared
inputs from «infra». Bundled linux binaries: blastn/makeblastdb 2.10.0,
fastq-dump 3.0.1.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
This reproduction of CRISPRbuilder-TB is in progress, not complete: the exact public SRA run (SRR6407486) and the authors' own deterministic tool are correctly set up, but every reproduced value (6-output run, ~100 bp read length, spoligotype octal 777777777760771) is still PENDING and agreement.json is not-run-yet. There is no fabrication signal and no observed deviation — the only gap is on our side (the SLURM reconstruction job had not finished). Because nothing has actually been compared, the core claim can be rated only limited/yellow; the inputs and endpoints are fully favorable (q1/q2 green) and should resolve cleanly once the deterministic run completes.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.