DeeReCT-APA: Prediction of Alternative Polyadenylation Site Usage Through Deep Learning.
The main results reproduced, with only marginal, non-material deviations.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No relevant deviation in data/preprocessing
- ✓No authors-side cause for any deviation
- ✓Any deviation was negligible
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
IN PROGRESS. DeeReCT-APA (deep learning, PyTorch) for alternative polyadenylation site usage prediction. Training/eval data ships in the repo (APA_ML/). Reproducing Table 1 (parental + F1: MAE, comparison accuracy, highest-usage accuracy, Spearman) via the authors' own code (main.py single-fold; cross_validation.py 5-fold) on «our HPC» GPU. Numbers TBD.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-18 ⛓ ce4bf19a847d
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-18
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-07-31
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan a deep learning model quantitatively predict the usage levels of all competing alternative polyadenylation sites (PASs) of a gene simultaneously, as a variable-length regression task that explicitly models inter-PAS competition, rather than as non-quantitative pairwise comparisons?
- ★ DeeReCT-APA, a CNN-LSTM deep learning architecture, quantitatively predicts the usage level of all alternative PASs within a gene regardless of PAS number, treating it as a variable-length regression task. method
- ★ Using a bidirectional LSTM as an interaction layer explicitly models the competition among multiple competing PASs of the same gene. method
- ★ DeeReCT-APA consistently outperforms existing methods (Polyadenylation Code and DeepPASTA) on the pairwise comparison task, highest usage prediction task, and ranking task. finding
- ★ Modeling the competition among multiple PASs simultaneously is necessary for accurate APA usage prediction. mechanism
- ★ DeeReCT-APA can predict the effect of genetic variations on APA patterns and visualize APA regulatory motifs. finding
- Among Base-Net designs, Multi-Conv-Net (multiple convolution layers) performs best, followed by Single-Conv-Net, then Feature-Net. finding
- Code and data are provided as an open resource at GitHub (lzx325/DeeReCT-APA-repo). resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Deep learning model training/evaluation (CNN-LSTM regression of PAS usage) | Genome-wide PAS dataset from mouse fibroblast cells (BL, SP strains and F1 hybrids) | none | Predicted percentage usage levels of all PASs per gene | PyTorch; NVIDIA GeForce GTX 980 Ti GPU |
| 3′-Region Extraction and Deep Sequencing (3′READS) | Fibroblast cell lines from BL and SP mouse strains | none | PAS reference of the two strains (PAS identification) | — |
| 3′-mRNA sequencing (PAS quantification) | Fibroblast cell lines from BL, SP, and F1 hybrid mice | none/genetic variation (strain-specific SNPs) | PAS usage values computed from sequencing reads assigned to each PAS | — |
- ▲ DeeReCT-APA with Multi-Conv-Net performs best among Base-Net designs on both parental BL and F1 datasets, followed by Single-Conv-Net.
- ▲ DeeReCT-APA with Multi-Conv-Net outperforms Polyadenylation Code and DeepPASTA on pairwise comparison, highest usage prediction, and ranking tasks.
- other 455-nt genomic DNA sequence input centered around each PAS cleavage site (Base-Net input sequence length)
- other 448 nt (sequence extracted centering around each PAS cleavage site for feature extraction/one-hot encoding)
- other 5% usage level difference threshold (minimum PAS pair usage difference retained for the pairwise comparison task)
- other 15% (minimum margin of highest over second-highest PAS usage for genes in the highest usage prediction task)
- count 5-fold cross validation at gene level (model evaluation scheme)
- other ~100 times smaller learning rate (fine-tuning on F1 dataset vs training from scratch)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
DeeReCT-APA is a CNN-BiLSTM regression model evaluated with 5-fold cross-validation (gene-level splits) on a genome-wide mouse fibroblast PAS usage dataset derived from two parental strains (C57BL/6J, SPRET/EiJ) and their F1 hybrids. Performance across three operationally distinct tasks—quantitative usage prediction, pairwise dominance classification, and within-gene PAS ranking—was quantified using MAE, classification accuracy, and averaged Spearman's rank correlation, respectively. Comparisons against two baseline methods (Polyadenylation Code, DeepPASTA) were made using the same cross-validation folds; no formal null-hypothesis tests or uncertainty estimates around metric values were reported.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Mean Absolute Error (MAE) | Quantitative PAS usage prediction task; primary regression evaluation metric for DeeReCT-APA variants | — | na |
| Pairwise comparison classification accuracy | Pairwise Comparison Task; predicting which PAS in an enumerated pair has higher usage, restricted to pairs with >5% usage difference | — | na |
| Highest usage prediction accuracy (classification accuracy) | Highest Usage Prediction Task; identifying the dominant PAS per gene, restricted to genes where the highest usage exceeds the second-highest by ≥15% | — | na |
| Averaged Spearman's rank correlation | Ranking Task; correlation between predicted and ground-truth PAS usage ranks computed per gene then averaged across all genes | — | not stated |
| 5-fold cross-validation (gene-level split) | Overarching evaluation framework applied to all metrics on BL, SP parental, and F1 hybrid datasets | — | not stated |
-
Cross-validation metrics (MAE, accuracy, Spearman correlation) were reported as single aggregate point estimates across 5 folds without fold-level variability↳ Could also: Report mean ± SD across folds, or apply a paired test (e.g., Wilcoxon signed-rank on per-fold metric differences) when comparing models — Fold-level variance quantifies whether observed differences between models are consistent or driven by one atypical fold, and a paired fold-level test would allow readers to assess whether numeric differences are statistically distinguishable beyond chance
-
Averaged Spearman's rank correlation was used as the within-gene ranking metric↳ Could also: Kendall's tau-b or normalized discounted cumulative gain (NDCG) could also measure within-gene PAS ranking quality — Kendall's tau has different sensitivity to tied ranks and may be more interpretable when per-gene PAS counts are small; NDCG weights errors at the top of the ranking more heavily, which may align better with biological interest in identifying the dominant PAS
-
MAE was used as the sole quantitative regression metric for PAS usage prediction↳ Could also: Root mean squared error (RMSE) or Pearson correlation between predicted and true usage fractions could also characterize regression performance — RMSE penalizes large prediction errors more strongly than MAE, which may matter if severely mispredicting a dominant PAS is more consequential than small uniform errors; Pearson correlation is widely used in comparable RNA quantification benchmarks and would facilitate cross-study comparison
-
A single 5-fold cross-validation was performed with a fixed random split↳ Could also: Repeated k-fold cross-validation (e.g., 10 repeats of 5-fold) or a fixed held-out test chromosome partition could also be used — Repeating the cross-validation reduces variance in performance estimates; a chromosome-held-out split would reduce spatial autocorrelation between train and test genes and would produce a benchmark more easily reproduced by future studies
-
The pairwise comparison task applied a single fixed usage-difference threshold (>5%) to filter ambiguous pairs before computing accuracy↳ Could also: Sensitivity analyses across multiple thresholds (e.g., 5%, 10%, 20%) or an area-under-threshold-curve summary could also characterize decision boundary behavior — A single threshold may favor methods that happen to perform best near that operating point; sweeping thresholds reveals whether accuracy advantages are robust across the full spectrum of easy-to-difficult pairs
-
Pre-trained parental models were fine-tuned on F1 hybrid data and evaluated within the same cross-validation scheme without a separate holdout for the fine-tuning step↳ Could also: A nested cross-validation or an explicit train/validation/test three-way split could also be used to evaluate fine-tuned models — When fine-tuning decisions (e.g., early stopping on validation MAE) and final evaluation share the same data partition, performance estimates can be optimistic; a nested or independent holdout design isolates fine-tuning generalization from the reported test-set estimate
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.