Interpretable prediction models for widespread m6A RNA modification across cell lines and tissues.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- ✓Overall, the reproduction was clean
- 🟡A deviation arose in the data or preprocessing
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough for a faithful 1:1 reproduction. CLSM6A ships its 11 per-cell-line pretrained CNNs + per-cell train/test splits. Loading the shipped models on the shipped independent test sets, with the authors' deployment-script dinucleotide-average (ENAC) encoding of the 201-nt window, reproduces the paper's two headline numbers essentially exactly: avg AUC 0.8047 vs reported 0.8040 (Δ0.0007) and avg AP 0.7831 vs 0.7811 (Δ0.0020). Exactly four cell lines exceed AUC 0.8 (matches paper), all three tissues exceed AP 0.874 (matches), and the reported 'rest' range bounds are reproduced to the digit by HCT116 (0.6090/0.6052) and HeLa (0.7271/0.7145). The one mismatch: HEK293T (AUC 0.7654) sits just above the stated 'rest' upper bound, a bucket-assignment difference that does not affect the averages. Diligence: a plain one-hot(201x4) encoding of the same models gives avg AUC 0.7479, confirming the dinucleotide encoding is the authors' true pipeline, not an assumption. NOT attempted (80/20): retraining from scratch, DeepLIFT/attention interpretability maps, baseline-method comparisons, and upstream GSE54365->m6A-Atlas site-calling. Per-cell-line Suppl. Table S4 exact numbers were not obtainable from the open text, so grading is against the main-text averages+ranges.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 79assessed: 2026-06-15 ⛓ e5d8bc8e8df1
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-15
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan a deep learning approach predict single-nucleotide-resolution m6A RNA modification sites across a broad range of human cell lines and tissues while also providing interpretable insight into the prediction decision-making process?
- ★ CLSM6A, a CNN-based model set, predicts single-nucleotide-resolution m6A RNA modification sites across eight cell lines and three tissues in H. sapiens resource
- ★ CLSM6A achieves superior performance compared to current state-of-the-art methods (im6A-TS-CNN and TS-m6A-DL) finding
- ★ CLSM6A interprets its prediction process by excavating critical filter-activated motifs and pinpointing important positions via forward and backward propagation method
- ★ Nucleotides at proximal positions surrounding the modification site contribute more to classification mechanism
- ★ Learned motifs significantly match known motifs, and highly activated motifs are strongly associated with high-impact motifs finding
- CLSM6A uses ENAC encoding (sequence length 2, sliding-window size 1) to represent input RNA sequences method
- CLSM6A exhibits better portability on similar cross-cell line/tissue datasets finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Computational m6A site prediction (CNN deep learning) | 8 human cell lines (A549, CD8T, HCT116, HEK293, HEK293T, HeLa, HepG2, MOLM13) and 3 tissues (brain, kidney, liver) | none | probability of m6A modification at single-nucleotide resolution; AUC, AP, MCC, Acc, Spe, Sen | PyTorch v1.7.1, NVIDIA RTX 3090 GPU |
| Model-based interpretation (filter/motif extraction, virtual pruning) | CLSM6A first convolutional layer filters on testing datasets | filter nullification (set parameters to zero) | activated amount and impact score; PWM/PFM sequence logos | R package ggseqlogo |
| Propagation-based interpretation (in silico mutagenesis, forward propagation) | CLSM6A models on input RNA sequences | single-nucleotide substitution to 3 alternative bases | attribution score / 4×l attribution map | — |
| Propagation-based interpretation (back-propagation, backward propagation) | CLSM6A models on input RNA sequences | none | derivative-based attribution map of size 4×l | — |
| Motif discovery | learned sequences from CLSM6A | none | discovered motifs matched to known motifs | DREME version 5.5.1 |
| Feature space visualization | last-layer features of testing-set positive/negative samples across cell lines | none | 2D embedding separating methylated vs non-methylated sites | t-SNE |
- ▲ CLSM6A achieved the best average performance across all evaluation metrics on independent test datasets AUC 0.8040, AP 0.7811
- ▲ CLSM6A reached the highest AUC in 10 of 11 cell lines/tissues 10 of 11
- ▲ AUC increased in A549, CD8T and MOLM13 over compared methods 4.27%–6.81%
- ▲ AUC and AP values on the three tissues were high AUC >0.912, AP >0.874
- ▲ Four cell lines reached high AUC and AP AUC and AP >0.8
- – Remaining cell lines had lower AUC and AP values AUC 0.609–0.727, AP 0.605–0.714
- ▲ Four cell lines (MOLM13, A549, CD8T, HEK293) improved with longer flanking sequence then plateaued; 100 nt flanking (201 nt) adopted
- – CLSM6A separates methylated and non-methylated sites well in feature space, more discriminative for accurate cell-line models
- other AUC 0.8040 (best average AUC of CLSM6A across 11 datasets)
- other AP 0.7811 (best average AP of CLSM6A across 11 datasets)
- fold_change 4.27%–6.81% AUC increase (improvement in A549, CD8T, MOLM13)
- other AUC >0.912, AP >0.874 (performance on three tissues)
- other AUC 0.609–0.727, AP 0.605–0.714 (remaining (weaker) cell lines)
- count 10 of 11 (datasets where CLSM6A had highest AUC)
- count 44317 (positive samples in HEK293T dataset (largest))
- other 80% (CD-HIT redundancy removal sequence similarity threshold)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
CLSM6A is a CNN-based deep learning study benchmarking m6A RNA modification site prediction across 11 cell line/tissue-specific binary classification tasks in H. sapiens. Model performance was evaluated on fixed held-out independent test sets (9:1 random split) using threshold-dependent metrics (Acc, MCC, Sen, Spe) and threshold-independent metrics (AUC of ROC, average precision of PRC); 5-fold cross-validation on the training portion was used for hyperparameter selection only. Baseline comparisons against im6A-TS-CNN and TS-m6A-DL were made by retraining those models on identical datasets and reporting point-estimate metrics side-by-side without formal statistical tests of metric differences. Model interpretability was assessed via filter-based motif extraction (PWM/PFM, cross-referenced with DREME), in silico mutagenesis attribution maps, and backpropagation attribution maps.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| 5-fold cross-validation (model selection and hyperparameter tuning) | Training partition of each of the 11 cell line/tissue-specific CNN models | — | not stated |
| AUC (area under ROC curve) and AP (average precision under PRC) — threshold-independent classification evaluation | Independent test sets for all 11 cell lines/tissues; cross-method comparison in Supplementary Table S4 | — | not stated |
| MCC (Matthews Correlation Coefficient), accuracy, sensitivity, specificity — threshold-dependent classification evaluation | Independent test sets for all 11 cell lines/tissues (Fig. 2A, Supplementary Table S4) | — | not stated |
| DREME motif discovery (Version 5.5.1) | Subsequences activating CNN filters above threshold in testing datasets; cross-referenced with known m6A motifs | — | not stated |
| t-SNE dimensionality reduction and visualization | Last-layer feature representations for positive and negative samples in testing datasets (Fig. 2D, Supplementary Fig. S2) | — | na |
-
Method comparisons were made by reporting point-estimate metrics (AUC, AP, MCC) side-by-side across 11 datasets without formal inferential tests↳ Could also: DeLong's test for pairwise AUC comparison, or bootstrap resampling of test-set predictions to obtain CIs and p-values for metric differences — Formal tests would quantify whether observed differences exceed chance variation, which is particularly informative when improvements are modest (e.g., the reported 4.27–6.81% AUC gains in specific cell lines)
-
Cross-validation was used only for hyperparameter selection; independent test set performance was reported as a single point estimate with no variability measure↳ Could also: Report mean ± SD of evaluation metrics across the 5 CV folds, or bootstrap 95% CIs on test-set metrics — Variability estimates help distinguish stable performance differences from fold-to-fold fluctuations, which is especially relevant for smaller datasets such as Liver (n=1574) and Kidney (n=3402)
-
A single fixed 9:1 random train/test split was used per cell line/tissue for final performance evaluation↳ Could also: Repeated random splits (e.g., 10 × 9:1) or stratified repeated k-fold on the full dataset, with performance aggregated across splits — A single split may yield an optimistic or pessimistic estimate by chance; repeated splits produce a distribution of performance values and a more stable generalization estimate
-
Sequence redundancy reduction with CD-HIT at 80% identity was applied before random train/test splitting↳ Could also: Apply CD-HIT clustering first, then split so that no cluster spans both sets (cluster-aware split), or use a stricter identity threshold (e.g., 60%) — If sequences sharing >80% identity land in both partitions after random splitting, test-set estimates may reflect some training-set similarity; a cluster-aware split would provide a more conservative estimate of generalization to novel sequences
-
Filter 'activated amount' and 'impact score' for CNN motif interpretation are reported as summary values without uncertainty quantification↳ Could also: Bootstrap confidence intervals on impact scores, or permutation-based null distributions to establish significance thresholds for filter importance — Uncertainty bounds on filter importance would help distinguish consistently high-impact filters from those whose scores vary across random model initializations or test-set subsamples
-
Balanced datasets were constructed by sampling an equal number of negative DRACH sites to match positives↳ Could also: Evaluate on class-imbalanced sets reflecting the true transcriptome ratio, relying on metrics already selected (MCC, AP) that are robust to imbalance — Artificial balancing alters the prior probability seen by the model; reporting on realistic imbalanced sets alongside the balanced results would clarify expected real-world deployment performance
Result convergence & founder nodes
Findings this paper shares with others that ran a comparable experiment. A node’s strength is how many independent papers report it (replication breadth) — not how often it is cited, so a heavily-replicated but under-cited founder still stands out.
-
Remaining cell lines show lower m6A prediction performance (AUC 0.609–0.727, AP 0.605–0.714), reflecting cell-line-specific variabilityother human-cell-line mixed 2023×1papers★ This paper is the founder (earliest)
-
CLSM6A last-layer features separate methylated from non-methylated sites in 2D embedding space, with greater discriminability in high-accuracy cell-line modelsother human-cell-line 2023×1papers★ This paper is the founder (earliest)
-
CLSM6A achieves best average m6A site prediction performance across all evaluation metrics on independent test datasets (AUC 0.8040, AP 0.7811)other human-cell-line up 2023×1papers★ This paper is the founder (earliest)
-
CLSM6A m6A site prediction achieves high performance across brain, kidney, and liver tissues (AUC >0.912, AP >0.874)other human-tissue up 2023×1papers★ This paper is the founder (earliest)
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-37995291 (CLSM6A)
Paper: Interpretable prediction models for widespread m6A RNA modification across cell lines and tissues. Zhang et al., Bioinformatics 2023. Repo (authors' own, P16 own-code): https://github.com/zhangying-njust/CLSM6A Data: GEO GSE54365 (m6A-seq, A549) + m6A-Atlas-derived sites; the repo ships the processed per-cell-line train/test CSVs directly, so GEO re-download is not on the critical path for the prediction-performance reproduction.
Pipeline
CLSM6A = per-cell-line 1D-CNN (3 Conv1d blocks, 64ch, k=5, MaxPool2, Dropout0.2;
FC 1600->256->1 sigmoid). Input = 201-nt sequence centered on candidate adenine,
encoded as 200 dinucleotide-average 4-vectors (ENAC-style) per the deployment
script. One model per cell line/tissue (11 total). Pretrained weights shipped in
models/<cell>.pkl; independent test sets in data_train_test/<cell>_test.csv
(cols: chr,loc,strand,seq,label).
IN SCOPE (deterministic, data+models shipped)
- C1: per-cell-line independent-test AUC/AP for all 11 cell lines/tissues by running the shipped pretrained model on the shipped test CSV. Compare to paper: avg AUC 0.8040 / avg AP 0.7811; 3 tissues AUC>0.912 & AP>0.874; 4 cell lines AUC>0.8; remaining cell lines AUC 0.609-0.727, AP 0.605-0.714 (Fig 2A / Suppl. Table S4). Also ACC/SN/SP/MCC at threshold 0.5.
OUT OF SCOPE / not attempted (80/20)
- Re-training the CNNs from scratch (stochastic; weights already shipped — we validate the shipped models instead).
- Interpretability outputs (DeepLIFT attribution maps, attention motifs) — these are qualitative figure outputs, not pinnable numeric claims.
- Comparison-method baselines (other m6A predictors) — external tools.
- Raw GSE54365 -> site-calling preprocessing (m6A-Atlas pipeline) — upstream of the shipped processed data; not the headline computational claim.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
This is an essentially exact reproduction: loading the authors' shipped per-cell-line CNNs on their shipped independent test splits with the confirmed dinucleotide-average encoding reproduces avg AUC 0.8047 vs 0.8040 and avg AP 0.7831 vs 0.7811, with the 'rest' range bounds matching to the digit. The only deviation is HEK293T landing just above the stated 'rest' bucket (AUC 0.7654 vs ≤0.727), a sample/bucket-assignment quirk on the input side that does not affect the headline averages. No fabrication signal and the central conclusion holds fully; deviations are negligible and technical.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.