DeeReCT-APA: Prediction of Alternative Polyadenylation Site Usage Through Deep Learning.
The main results reproduced, with only marginal, non-material deviations.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No relevant deviation in data/preprocessing
- ✓No authors-side cause for any deviation
- ✓Any deviation was negligible
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
IN PROGRESS. DeeReCT-APA (deep learning, PyTorch) for alternative polyadenylation site usage prediction. Training/eval data ships in the repo (APA_ML/). Reproducing Table 1 (parental + F1: MAE, comparison accuracy, highest-usage accuracy, Spearman) via the authors' own code (main.py single-fold; cross_validation.py 5-fold) on «our HPC» GPU. Numbers TBD.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-18 ⛓ ce4bf19a847d
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-18
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-07-31
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetCan a deep learning model that jointly considers all competing polyadenylation sites (PASs) of a gene—rather than comparing them only pairwise—more accurately and quantitatively predict the usage levels of alternative PASs?
- ★ DeeReCT-APA quantitatively predicts the usage of all competing PASs of a gene simultaneously, rather than casting the problem as pairwise comparison like prior methods. method
- ★ DeeReCT-APA uses a CNN-LSTM architecture with a bidirectional LSTM (BiLSTM) interaction layer to explicitly model competition among alternative PASs. method
- ★ DeeReCT-APA consistently outperforms Polyadenylation Code and DeepPASTA on the pairwise comparison task, highest usage prediction task, and ranking task. finding
- ★ Explicitly modeling competition among multiple PASs simultaneously (via the BiLSTM interaction layer) improves performance relative to ablated versions without RNN interaction or with comparison-based training. finding
- ★ DeeReCT-APA can be used to predict the effect of genetic variation (cis-regulatory sequence differences between mouse strains) on APA patterns. finding
- Among the three Base-Net designs, Multi-Conv-Net performs best, followed by Single-Conv-Net, then Feature-Net, on both parental and F1 datasets. finding
- ★ Unlike DeepPASTA's BiLSTM, which processes sequence positions within a single PAS sequence, DeeReCT-APA's BiLSTM time steps correspond to different competing PASs, directly modeling PAS-PAS interactions. mechanism
- Code and data for DeeReCT-APA are publicly available on GitHub. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| 3'-Region Extraction and Deep Sequencing (3'READS) | fibroblast cells, BL (C57BL/6J) and SP (SPRET/EiJ) mouse strains | none (natural strain genetic variation) | genome-wide reference of polyadenylation sites (PASs) | — |
| 3'-mRNA sequencing | fibroblast cells of BL, SP, and F1 hybrid mice | none / natural cis-regulatory genetic variation between alleles | PAS usage level (sequencing reads assigned to each PAS, allele-specific in F1) | — |
| deep learning regression model (DeeReCT-APA CNN-LSTM) training with 5-fold cross-validation | mouse genomic DNA sequences (455-nt/448-nt regions flanking PAS cleavage sites) | none | predicted percentage usage of each competing PAS in a gene | PyTorch; NVIDIA GeForce GTX 980 Ti GPU |
| ablation: BiLSTM interaction layer removed (fully connected + SoftMax only) | same parental/F1 PAS dataset | architectural ablation | usage prediction performance (mean absolute error) | — |
| ablation: interaction layer removed entirely, comparison-based training (as in Polyadenylation Code) | same parental/F1 PAS dataset | architectural ablation | comparison accuracy | — |
| Pairwise Comparison Task (benchmark vs Polyadenylation Code, DeepPASTA) | same parental (BL, SP) and F1 PAS dataset | none | comparison accuracy (correctly predicted higher-usage PAS in pair) | — |
| Highest Usage Prediction Task | same parental and F1 PAS dataset | none | accuracy of predicting the PAS with highest usage per gene | — |
| Ranking Task | same parental and F1 PAS dataset | none | averaged Spearman's rank correlation between predicted and ground-truth PAS usage ranking | — |
- ▲ DeeReCT-APA with Multi-Conv-Net Base-Net achieves the best performance among the three Base-Net designs (Multi-Conv-Net > Single-Conv-Net > Feature-Net) on both the parental BL dataset and the F1 dataset
- other 455 nt (length of genomic DNA sequence input to each Base-Net, centered on a PAS cleavage site)
- other 448 nt (total length of sequence extracted around each PAS cleavage site for feature extraction/one-hot encoding)
- other 5-fold cross validation (gene-level cross-validation scheme used to evaluate DeeReCT-APA)
- other >5% usage difference threshold (minimum PAS usage-level difference required for a pair to be included in the Pairwise Comparison Task)
- other ≥15% difference between highest and second-highest PAS usage (gene selection threshold for the Highest Usage Prediction Task)
- other ~100-fold smaller learning rate (learning rate reduction applied when fine-tuning the parental model on the F1 dataset)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
DeeReCT-APA is a CNN-BiLSTM regression model evaluated with 5-fold cross-validation (gene-level splits) on a genome-wide mouse fibroblast PAS usage dataset derived from two parental strains (C57BL/6J, SPRET/EiJ) and their F1 hybrids. Performance across three operationally distinct tasks—quantitative usage prediction, pairwise dominance classification, and within-gene PAS ranking—was quantified using MAE, classification accuracy, and averaged Spearman's rank correlation, respectively. Comparisons against two baseline methods (Polyadenylation Code, DeepPASTA) were made using the same cross-validation folds; no formal null-hypothesis tests or uncertainty estimates around metric values were reported.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Mean Absolute Error (MAE) | Quantitative PAS usage prediction task; primary regression evaluation metric for DeeReCT-APA variants | — | na |
| Pairwise comparison classification accuracy | Pairwise Comparison Task; predicting which PAS in an enumerated pair has higher usage, restricted to pairs with >5% usage difference | — | na |
| Highest usage prediction accuracy (classification accuracy) | Highest Usage Prediction Task; identifying the dominant PAS per gene, restricted to genes where the highest usage exceeds the second-highest by ≥15% | — | na |
| Averaged Spearman's rank correlation | Ranking Task; correlation between predicted and ground-truth PAS usage ranks computed per gene then averaged across all genes | — | not stated |
| 5-fold cross-validation (gene-level split) | Overarching evaluation framework applied to all metrics on BL, SP parental, and F1 hybrid datasets | — | not stated |
-
Cross-validation metrics (MAE, accuracy, Spearman correlation) were reported as single aggregate point estimates across 5 folds without fold-level variability↳ Could also: Report mean ± SD across folds, or apply a paired test (e.g., Wilcoxon signed-rank on per-fold metric differences) when comparing models — Fold-level variance quantifies whether observed differences between models are consistent or driven by one atypical fold, and a paired fold-level test would allow readers to assess whether numeric differences are statistically distinguishable beyond chance
-
Averaged Spearman's rank correlation was used as the within-gene ranking metric↳ Could also: Kendall's tau-b or normalized discounted cumulative gain (NDCG) could also measure within-gene PAS ranking quality — Kendall's tau has different sensitivity to tied ranks and may be more interpretable when per-gene PAS counts are small; NDCG weights errors at the top of the ranking more heavily, which may align better with biological interest in identifying the dominant PAS
-
MAE was used as the sole quantitative regression metric for PAS usage prediction↳ Could also: Root mean squared error (RMSE) or Pearson correlation between predicted and true usage fractions could also characterize regression performance — RMSE penalizes large prediction errors more strongly than MAE, which may matter if severely mispredicting a dominant PAS is more consequential than small uniform errors; Pearson correlation is widely used in comparable RNA quantification benchmarks and would facilitate cross-study comparison
-
A single 5-fold cross-validation was performed with a fixed random split↳ Could also: Repeated k-fold cross-validation (e.g., 10 repeats of 5-fold) or a fixed held-out test chromosome partition could also be used — Repeating the cross-validation reduces variance in performance estimates; a chromosome-held-out split would reduce spatial autocorrelation between train and test genes and would produce a benchmark more easily reproduced by future studies
-
The pairwise comparison task applied a single fixed usage-difference threshold (>5%) to filter ambiguous pairs before computing accuracy↳ Could also: Sensitivity analyses across multiple thresholds (e.g., 5%, 10%, 20%) or an area-under-threshold-curve summary could also characterize decision boundary behavior — A single threshold may favor methods that happen to perform best near that operating point; sweeping thresholds reveals whether accuracy advantages are robust across the full spectrum of easy-to-difficult pairs
-
Pre-trained parental models were fine-tuned on F1 hybrid data and evaluated within the same cross-validation scheme without a separate holdout for the fine-tuning step↳ Could also: A nested cross-validation or an explicit train/validation/test three-way split could also be used to evaluate fine-tuned models — When fine-tuning decisions (e.g., early stopping on validation MAE) and final evaluation share the same data partition, performance estimates can be optimistic; a nested or independent holdout design isolates fine-tuning generalization from the reported test-set estimate
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at [email protected].
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.