Systematic clustering algorithm for chromatin accessibility data and its application to hematopoietic cells.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No relevant deviation in data/preprocessing
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- ✓Overall, the reproduction was clean
- Every checked point held up.
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
1:1 REPRODUCED via the authors' own tool on the authors' shipped data (P16-valid). Described well enough: the repo ships the 77-sample (13 cell-type) preprocessed reference (top-64000-peak BEDs + 77x77 Hamming matrix) and a CLL test sample, and the README documents the exact run. The headline numeric claim reproduces EXACTLY: clustering penalty = 18 at the optimal M_cut=64000 (authors' unmodified penaltyscore.py on the Ward linkage of the shipped matrix). The shipped CLL test sample is classified closest to B cells (exact, matching Table 3's 7/7 CLL->B). The 77-sample Ward dendrogram recovers all 13 cell types as coherent branches with the known HSC/MPP overlap that the penalty=18 quantifies. A fabrication check (C4) shows the shipped distance matrix is provably derivable from the shipped per-sample BEDs (6/6 sampled pairs exact, both directions) -> no fabrication signal. Honest deviations (documented, none affecting the numbers): scripts run under py3 instead of py2.7 (README permits py3); the only ready py3.9 env had ABI-broken pandas/scipy, so the TSV was read with numpy and dendrograms drawn with scipy.dendrogram while the numeric core (scipy ward linkage + authors' unmodified penaltyscore.py/ward_closest.py + bedtools Hamming distances) was unchanged. NOT attempted (out of scope): raw-read pipeline from SRA, M_cut optimization sweep, Fig 12 ChromHMM crossovers, Fig 13 UPGMA=36, Fig 14 robustness sweep, and Tables 4-5 AML/ATL patient case analyses (patient samples not shipped). All grades provisional; a human reviewer signs off.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 90assessed: 2026-06-18 ⛓ 7f153605dc64
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-18
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-18no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan a systematic clustering algorithm that binarizes genome-wide open-chromatin (ATAC-seq) data and uses Hamming distances classify hematopoietic cell types more efficiently than existing read-count/quantile-normalization approaches, and thereby yield insights into leukemia?
- ★ Representing the genome as a binary string (1/0) of peak presence and computing Hamming distances enables hierarchical clustering of ATAC-seq samples with reduced computational cost while preserving cell-type classification. method
- ★ Systematically optimizing the peak-selection parameters (M_cut, p_G) by minimizing a clustering 'penalty' cost function yields the best cell-type classification dendrogram. method
- ★ The optimized binarization/Hamming-distance clustering achieves a lower classification penalty than a previous read-count-based method. finding
- ★ ATAC-seq peak width decreases statistically as peak p-value increases, justifying robust data reduction by selecting low-p-value peaks. finding
- The algorithm can infer the phenotype of a leukemia sample (CLL, AML, ATL) using only that sample's ATAC-seq data as input. method
- The findclosest software implementing the algorithm is provided as a resource. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| ATAC-seq (chromatin accessibility) | 13 human primary blood/hematopoietic cell types (HSC, MPP, LMPP, CMP, MEP, GMP, CLP, NK, B, CD4+ T, CD8+ T, Mono, Ery), 77 samples | none | genome-wide open-chromatin peaks binarized to 1/0 strings; Hamming distances for clustering | MACS2 peak calling on hg19 reference; reads MQ>=30 |
| ATAC-seq (leukemia application) | human leukemia cells: chronic lymphocytic leukemia (CLL), acute myeloid leukemia (AML), adult T-cell leukemia (ATL) | none | inferred cell phenotype from chromatin accessibility | — |
| RNA-seq | hematopoietic/leukemia samples (DDBJ DRA010939) | none | gene expression (deposited to reproduce study) | — |
- – Optimized clustering parameters (M_cut*, p_G*) found near (64000, 10^-2), giving the smallest penalty in the search resolution (64000, 10^-2)
- ▼ Minimum global penalty achieved by the new method at optimal parameters penalty = 18 (vs 27 for prior read-count method)
- – Estimated optimal p_G range from no-cutoff analysis 1.5 <= -log10(p_G*) <= 4
- ▼ ATAC-seq peak width becomes statistically shorter as peak p-value increases
- count 77 ATAC-seq datasets (samples from 13 human primary blood cell types used as test data)
- other penalty = 18 (minimum global penalty of new method at (M_cut, p_G)=(64000, 10^-2))
- other penalty = 27 (penalty of prior clustering using GSE74912 read-count data)
- other (64000, 10^-2) (best parameters (M_cut*, p_G*); 64000 is midpoint of 60000-68000 giving same minimum penalty)
- pvalue 10^-20.36428 (example Mono peak p-value at (1092756,1094068) for p_G=10^-2)
- other 906 and 676 bp (widths of left/right NK-cell peaks on chr3)
- other 1.5 <= -log10(p_G*) <= 4 (concluded range for optimal p_G with M_cut=infinity)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper develops a custom ATAC-seq sample-classification algorithm that binarizes genome-wide MACS2 peak calls into per-sample 1/0 genomic strings, computes pairwise Hamming distances between these strings, and performs agglomerative hierarchical clustering using Ward's linkage. Two algorithmic parameters — a peak-rank cutoff (M_cut) and a peak-calling p-value threshold (p_G) — are jointly optimized by grid search minimizing a custom 'global penalty' (λ) that counts immunophenotypically misclassified samples at each dendrogram merge node. Clustering performance is reported as the achieved minimum integer penalty score, compared directly against a previously published pipeline on the same 77-sample dataset.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Hierarchical clustering, Ward's minimum-variance linkage (agglomerative) | All 77 ATAC-seq samples across 13 hematopoietic cell types; produces the classification dendrogram | 77 samples | not stated |
| Hamming distance (pairwise binary string dissimilarity) | All pairwise combinations of 77 samples; input distance matrix for hierarchical clustering | 77 samples | not stated |
| Custom global penalty function minimization (grid search over M_cut × p_G parameter space) | Parameter optimization selecting (M_cut*, p_G*) = (64000, 10^-2) yielding λ = 18 vs. λ = 27 for prior method | 77 samples | not stated |
| MACS2 peak calling (negative-binomial/Poisson-based p-value thresholding) | All 77 ATAC-seq samples individually for open-chromatin peak detection; p_G threshold used as an optimization parameter | per-sample read depth (not stated numerically) | not stated |
-
Binary peak presence/absence (0/1) was used to represent chromatin accessibility per sample, discarding within-peak read-count magnitude↳ Could also: Continuous read-count matrices (normalized by TMM, DESeq2 size factors, or RPKM/CPM) combined with PCA- or SVD-based dimensionality reduction could also represent chromatin accessibility for clustering — Continuous representations retain quantitative depth information; the authors explicitly motivate binarization as a way to avoid cross-sample quantile normalization assumptions when samples are phenotypically heterogeneous, a trade-off worth making transparent when comparing approaches
-
Pairwise Hamming distance was used as the dissimilarity measure between binary genome strings↳ Could also: The Jaccard similarity coefficient or Sørensen–Dice coefficient could also quantify similarity between two binary peak sets — Jaccard and Dice normalize by the union or sum of set cardinalities and are therefore less sensitive to genomic positions where peaks are universally absent in all samples; choice of binary distance metric can affect dendrogram topology
-
Ward's minimum-variance criterion was used as the sole linkage method for agglomerative hierarchical clustering↳ Could also: Complete linkage, average linkage (UPGMA), or non-hierarchical methods (k-medoids, UMAP + HDBSCAN) could also group samples from the same Hamming-distance matrix — Different linkage criteria yield dendrograms with different compactness and topology; comparing multiple linkage methods is a standard robustness check in genomic clustering studies
-
Clustering quality was evaluated by a custom penalty function λ that counts immunophenotypically misclassified samples at each cell-type merge node↳ Could also: Standard external cluster-validity indices such as the Adjusted Rand Index (ARI) or Normalized Mutual Information (NMI), computed against the known immunophenotype labels, could also serve as evaluation metrics — ARI and NMI are widely adopted, interpretable benchmarks that correct for chance agreement and enable direct comparison across methods and datasets beyond this study
-
Dendrogram branch stability was assessed only implicitly through the penalty-score landscape across the parameter grid↳ Could also: Bootstrap resampling approaches (e.g., pvclust in R) could also attach approximately-unbiased (AU) probabilities to individual dendrogram branches — Branch-level confidence estimates distinguish robust splits from sampling-sensitive ones, and are frequently reported alongside hierarchical clustering in genomics studies to support interpretation of specific groupings
-
Parameter optimization (M_cut, p_G) and performance evaluation both used the same 77 samples, with performance reported at the single optimized point↳ Could also: Leave-one-out or k-fold cross-validation of the penalty score could also estimate how well the optimized parameters generalize to new samples — Evaluating on the same data used for optimization may overstate generalizability; a held-out or cross-validated penalty estimate would provide an additional perspective on whether the chosen parameters are stable across sample subsets
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-33253153
Paper: Tanaka A, Ishitsuka Y, Ohta H, Fujimoto A, Yasunaga JI, Matsuoka M. "Systematic clustering algorithm for chromatin accessibility data and its application to hematopoietic cells." PLoS Comput Biol 2020;16(12):e1008422. PMID 33253153 · PMCID PMC7728210 · DOI 10.1371/journal.pcbi.1008422.
Code: https://github.com/tanakanishi/findclosest (authors' own tool, default
branch master, HEAD SHA 82ef634cd604672e44ffac1a733bbf30fb6053cf).
Data: GEO GSE74912 (Corces et al. hematopoiesis ATAC-seq). The tool ships a
77-sample subset of GSE74912 already preprocessed into top-64000-peak BED
files plus a precomputed 77×77 Hamming-distance matrix.
This is a P16-valid reproduction: we run the authors' shipped tool on the authors' shipped data exactly as the README documents.
Method (what the tool does)
- Binarization / top-peak selection. Each sample's MACS2 narrowPeak is
ranked by ascending p-value and the top M_cut = 64000 peaks are kept as
genomic intervals (
*.64000.bed). Binary "open=1/closed=0" over the genome. - Hamming distance. Between two samples = total bp in the symmetric
difference of their peak intervals, computed via
bedtools subtractBed+distance.py(sum of interval lengths), summed both directions (readfile2.sh). Yields a sample×sample distance matrix. - Ward hierarchical clustering of that matrix (
scipy ... method="ward"), drawn as a colored dendrogram (ColoredClustering.py). - Closest-cell-type assignment for a query sample (
ward_closest.py): per-cell-type sum-of-squares score over the query's distances to that type's reference members, minus a per-type constant; smallest = closest. - Penalty score (
additional/penaltyscore.{sh,py}): walks the Ward merge tree of the 77 reference samples and counts mis-grouped samples at the point each of the 13 cell-type clusters first forms; lower = better clustering.
IN SCOPE (pipeline-derived, attempted)
| id | result | reported | pipeline | inputs (all shipped) |
|---|---|---|---|---|
| C1 | Global penalty of the 77-sample reference clustering at M_cut=64000 | 18 (Fig 10/11; "previous method" = 27, UPGMA = 36) | penaltyscore.sh pvalue2_64000_77samples.txt |
additional/ (shipped 77×77 matrix) |
| C2 | Closest cell type for the CLL test sample | Bcell (rank 1) — CLL is a B-cell malignancy; Table 3: 7/7 CLL→B | calculate.sh test1.narrowPeak → ward_closest.py |
test1.narrowPeak + GEOdata/ 77 beds + merge matrix |
| C3 | 77-sample Ward dendrogram recovers the 13 hematopoietic cell types as coherent branches | qualitative (main clustering figure) | ColoredClustering.py pvalue2_64000_77samples.txt |
shipped 77×77 matrix |
| C4 | Internal consistency: distances recomputed from the shipped GEOdata BEDs match the shipped 77×77 matrix (fabrication check) | — (self-check) | subtractBed+distance.py on GEOdata beds vs merge matrix |
GEOdata/ + merge matrix |
| C5 | Dataset N: shipped reference = exactly 77 samples / 13 cell types | "77 ATAC-seq datasets from 13 ... cell types" (Abstract/Methods) | count GEOdata beds + colname/samplelist | repo metadata |
OUT OF SCOPE (not attempted, with reason)
- Raw read processing (bwa → samtools MQ≥30 → PICARD dedup → MACS2 peak calling) of the GSE74912 FASTQ/BAM: the tool ships the already-peak-called top-64000 BEDs; raw reads are not redistributed by the tool and re-deriving them from SRA is a separate heavy pipeline not needed to reproduce the tool's documented outputs. (Could be added later; large.)
- M_cut optimization sweep / Fig 12 functional-annotation crossovers (ChromHMM enhancer/promoter), Fig 13 UPGMA=36, Fig 14 robustness (r-data-loss): require regenerating per-M_cut matrices and external ChromHMM annotations not shipped; deferred (harder, beyond the quick floor).
- Leukemia case analyses (Table 4 AML pHSC 12/16, Blast 13/1
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
This is a clean 1:1 reproduction: the authors' own unmodified tool, run on their shipped preprocessed reference data, yields the headline clustering penalty exactly = 18 at M_cut=64000 (Figs 10-11), and the shipped CLL test sample classifies closest to Bcell, matching Table 3's 7/7. An explicit fabrication check (C4) shows the shipped 77x77 Hamming matrix is provably derivable from the shipped BEDs (6/6 pairs exact), so there is no too-perfect signal. The only deviations are on our side and immaterial (py2→py3, scipy.dendrogram swapped for the authors' ABI-broken plotter, <1 ulp float division), and the sole qualitative residual (HSC/MPP overlap in C3) is exactly what penalty=18 quantifies. The central conclusion holds fully; reproduction quality is green.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.