Adaptive learning embedding features to improve the predictive performance of SARS-CoV-2 phosphorylation sites.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- Nothing in this column.
- 🔴Could not use the authors’ exact input data
- 🔴Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🔴Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🔴Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
PSPred-ALE (SARS-CoV-2 phospho-site predictor) is described well enough to RUN its shipped artifact but NOT to reproduce its headline numbers. The repo (commit 76413c8) and Zenodo 8330277 (same zip) ship a trained PyTorch model + inference-only main.py + a 4-sequence demo with expected outputs, plus unrelated DESeq2 gene-DE tables. They do NOT ship the labeled benchmark (5387 pos + 5387 neg S/T phospho-peptides), the 80/20 split, or any training script. REPRODUCED 1:1: in the paper's exact env (Python 3.8 / PyTorch 1.13.1 / NumPy 1.24.2 / Pandas 1.5.3, rebuilt on «our HPC»), running main.py on test.fasta reproduced Results/Results.csv byte-for-byte (deterministic eval) -> the model + inference pipeline are genuine and reproducible (grade exact). NOT ATTEMPTED: Table 1 metrics (independent test ACC 83.14% / MCC 0.663 / AUC 0.907; 5-fold CV ACC 84.03% / MCC 0.681 / AUC 0.913) - there is no labeled input to recompute Sn/Sp/ACC/MCC/AUC on and no training code/seed to regenerate the split; reconstructing the benchmark from the upstream phosphoproteome is the hard >20% and would not match 1:1, so per the brief it was not chased. FABRICATION FLAG: the Table 1 headline numbers are not independently derivable from the deposited artifacts (model present, but benchmark labels + split + training code absent) - flagged for the human auditor, not asserted as fabrication. Env gotcha: pytorch 1.13.1 needs mkl pinned to 2023.1.0 (mkl-2026 drops iJIT_NotifyEvent).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 67assessed: 2026-06-15 ⛓ 5d22eb312ec8
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-15
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan a self-adaptive learning embedding deep-learning model (PSPred-ALE) that automatically extracts contextual sequence features outperform existing tools at identifying SARS-CoV-2-associated phosphorylation sites in infected human host cells, and does SARS-CoV-2 infection measurably alter host-cell phosphorylation pathways?
- ★ PSPred-ALE, a deep learning predictor using a self-adaptive learning embedding algorithm, automatically extracts contextual sequence features and identifies SARS-CoV-2 phosphorylation sites without feature engineering. method
- ★ Self-adaptive learning embedding features are superior to hand-crafted statistical features in capturing discriminative sequence information. finding
- ★ PSPred-ALE outperforms state-of-the-art SARS-CoV-2 phosphorylation site predictors and handcrafted feature-based methods. finding
- ★ A multihead self-attention module captures global information in protein sequences, further improving prediction accuracy. method
- ★ SARS-CoV-2 infection of A549 cells produces dramatic changes in host-cell protein phosphorylation pathways, becoming significant 12 and 24 h post infection. mechanism
- PSPred-ALE is an open, end-to-end tool available on GitHub and Zenodo. resource
- The transcriptional response to the virus peaked at 12 h post infection and increased steadily to 24 h post infection. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| bulk RNA-seq / transcriptomic bioinformatic analysis (DEG, KEGG/GO, GSEA, PCA) | human lung epithelial A549 cells infected with SARS-CoV-2 (USA-WA1/2020, MOI 2) at 2/6/12/24 h | SARS-CoV-2 infection vs uninfected control | differentially expressed genes, pathway enrichment, principal components | GEO dataset GSE184536; R/limma, R/clusterProfiler, R/biomaRt, R/GenePlot2 |
| Deep learning phosphorylation-site prediction (PSPred-ALE: embedding + multihead attention) | peptide sequences from human A549 cells infected with SARS-CoV-2 | none (in silico classification) | binary classification of S/T residue as phosphorylated vs non-phosphorylated; ACC, SE, SP, AUC, MCC | — |
| Traditional machine learning classification with 20 handcrafted feature encodings | same benchmark peptide dataset (S/T-centered, length 33) | none (in silico) | classification performance metrics (ACC, SE, SP, AUC, MCC) | iLearn package (feature extraction), scikit-learn API (SVM, RF, LGBM, XGBT) |
- – First two principal components captured 92.49% of total variance in the mRNA dataset (PC1 87%, PC2 5.49%), separating infected from uninfected groups. 92.49% (PC1 87%, PC2 5.49%)
- – Protein phosphorylation pathways showed no significant change at 2 and 6 h but significant changes at 12 and 24 h post infection.
- – Immune-related pathways were highly enriched at 12 and 24 h post infection (KEGG); phosphorylation-related pathways more enriched in uninfected than infected groups (GO).
- ▲ The only prior dedicated predictor, DeepIPs, achieves about 80% accuracy, which PSPred-ALE exceeds. ~80% (DeepIPs baseline)
- ▲ Self-adaptive learning embedding features outperformed handcrafted statistical features in 2D visualization separation of positive/negative samples.
- count 14119 experimentally verified phosphorylation sites collected (sites identified in SARS-CoV-2-infected human A549 cells)
- count 10774 final samples (5387 positive + 5387 negative) (balanced benchmark dataset for model training/testing)
- other CD-HIT identity threshold 0.3; peptide segment length 33 (redundancy reduction and sequence windowing)
- other 80%/20% train-validation vs independent blind test split (dataset partitioning)
- pvalue P<.01 and |LogFC>0.1| for DEGs; P<.05 for enrichment/GSEA (thresholds for DEG and pathway enrichment significance)
- other PCA variance 92.49% (PC1 87%, PC2 5.49%) (variance explained by first two principal components)
- count Top 26 KEGG pathways and top 25 GO pathways shown (heatmap enrichment results across infection time course)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper combines bioinformatics analysis of RNA-seq data from SARS-CoV-2-infected A549 cells — using limma-based DEG analysis, GO/KEGG enrichment, and GSEA — with the development and benchmarking of a deep learning phosphorylation-site predictor (PSPred-ALE). For the machine-learning component, a balanced benchmark dataset of 10,774 peptide sequences is split 80/20 into training and an independent blind test set. Model performance is reported as point estimates of ACC, SE, SP, AUC, and MCC, with comparative analysis against 20 handcrafted feature encodings paired with four classical classifiers (SVM, RF, LGBM, XGBT).
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| limma linear model / moderated t-statistic (DEG analysis) | Differentially expressed genes in SARS-CoV-2-infected vs uninfected A549 cells (RNA-seq, GSE184536) | Independent biological triplicates per condition and time point | not stated |
| GO enrichment analysis (hypergeometric/Fisher test via R/clusterProfiler) | Biological process enrichment across infection time points, Fig. 2C heatmap | — | not stated |
| KEGG pathway enrichment analysis (via R/clusterProfiler) | Pathway enrichment at 2/6/12/24 h post-infection, Fig. 2B heatmap | — | not stated |
| Gene Set Enrichment Analysis (GSEA) | Viral infection group vs control groups for protein phosphorylation pathway changes, Fig. 2D–G | Independent biological triplicates per time point (2/6/12/24 h post-infection) | not stated |
| Principal Component Analysis (PCA) | Dimensionality reduction and visualization of mRNA dataset across infected and uninfected conditions, Fig. 2A | Independent biological triplicates per condition/time point | na |
-
Class imbalance was addressed by randomly undersampling the majority (negative) class to match the positive count, yielding 5,387 samples per class↳ Could also: Oversampling methods such as SMOTE, or cost-sensitive learning with class weights, could also be applied to handle imbalance — SMOTE generates synthetic minority-class examples rather than discarding majority-class data, retaining more training information; cost-sensitive learning avoids any data manipulation and is natively supported by the classifiers used (SVM, RF, LGBM, XGBT)
-
ML model performance was estimated on a single fixed 80/20 train/test split↳ Could also: Stratified k-fold cross-validation (e.g., 5- or 10-fold) could also be used for performance estimation — Cross-validation produces multiple held-out estimates and a variance measure across folds, yielding a more stable generalization estimate from the same data without requiring a larger dataset
-
DEG analysis used R/limma with fixed thresholds of |LogFC| > 0.1 and P < .01 across thousands of genes↳ Could also: DESeq2 with a negative binomial model and built-in shrinkage estimation could also be applied to RNA-seq count data — DESeq2 is specifically designed for count-based RNA-seq and applies its own dispersion shrinkage and FDR control; the choice between limma-voom and DESeq2 is well-studied and each has scenarios where it performs favorably
-
Multiple statistical thresholds were applied across thousands of genes and gene sets without an explicit multiple-testing correction method being stated↳ Could also: Benjamini-Hochberg FDR correction could also be explicitly applied and reported for both DEG and enrichment analyses — With genome-scale testing, FDR control is a standard approach to limit the expected proportion of false positives among reported findings and is straightforward to implement in both limma and clusterProfiler
-
ML model performance metrics (ACC, AUC, MCC, etc.) were reported as single point estimates on the held-out blind test set↳ Could also: Bootstrapped 95% confidence intervals around AUC and MCC could also be reported — Confidence intervals quantify estimation uncertainty in performance metrics; with a test set of approximately 2,155 sequences, bootstrapping is computationally inexpensive and allows readers to assess whether observed differences between models exceed sampling variability
-
Comparisons across the 20 feature sets and 4 classifiers were based on reported metric values without a formal statistical test of performance differences↳ Could also: McNemar's test or DeLong's test for AUC comparison could also be used to formally assess whether performance differences between models on the same test set are statistically significant — Formal paired-prediction significance tests provide a principled basis for distinguishing true performance differences from sampling variability, especially when metric differences are small
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-37847658 (PSPred-ALE)
Paper: Jiao et al. 2023, Bioinformatics 39(11):btad627. "Adaptive learning
embedding features to improve the predictive performance of SARS-CoV-2
phosphorylation sites." PMID 37847658.
Code: https://github.com/jiaoshihu/PSPred-ALE @ commit 76413c87e8088808282c80177a2b64b7106f7907
Data: zenodo 10.5281/zenodo.8330277 — this is the v1.0.0 zip of the same
GitHub repo, NOT a separate dataset deposit.
What the artifact ships
data/model_saved.pkl(572 KB) — the trained PyTorch model state_dict.data/residue2idx.pkl(27 KB) — token→index map (vocab 3188; the "adaptive learning embedding" vocabulary).model/PSPred_model.py— Transformer encoder/decoder + multi-head attention, config: n_layers=1, n_heads=8, d_model=16, d_ff=32, tgt_len=29.preprocess/data_process.py— tokenize + pad to length 29.main.py— inference only: read FASTA → predict p-site / Non-p site + confidence.test.fasta— 4 demo sequences (length-29 peptides).Results/Results.csv— shipped expected predictions fortest.fasta.data/bioinformatics analysis data/— DESaq2-style gene-level differential expression tables (cols: baseMean, log2FoldChange, pvalue, padj) for the Mock-vs-SARS-CoV-2 time-course +sample class.txt. Downstream proteomics/ transcriptomics analysis, unrelated to the phospho-site classifier benchmark.
In scope (pipeline-derived, ML classifier)
- C1 — Inference-pipeline reproducibility (REPRODUCIBLE). The shipped
pretrained model +
main.pyrun deterministically (model.eval(), no dropout) and should reproduceResults/Results.csvbyte-for-byte ontest.fasta. This is the one clean, fully-specified, low-hanging pipeline output. Pipeline: PSPred-ALE PyTorch inference.
Out of scope / NOT reproducible — and why
- Table 1 metrics (independent test ψ=29: ACC 83.14 %, SE 83.15 %, SP 83.13 %,
MCC 0.663, AUC 0.907; 5-fold CV: ACC 84.03 %, MCC 0.681, AUC 0.913).
The labeled benchmark dataset is NOT deposited. The repo/Zenodo ship the
trained model but neither the positive/negative S/T phospho-peptide sets
(paper: 5387 + 5387 = 10 774 samples), the 80/20 train/test split, nor any
training script or split seed. Without the labeled independent-test peptides
there is no input on which to recompute Sn/Sp/ACC/MCC/AUC, and without training
code + seed the split cannot be regenerated. → not attempted (would require
reconstructing the benchmark from the upstream phosphoproteome = the hard
20%, and even then would not match 1:1).
bioinformatics analysis data/DE tables — wet-lab/proteomics downstream, not the classifier pipeline. Out of scope.
Possible-fabrication note (for the human auditor)
The Table 1 headline numbers are not independently derivable from the shipped artifacts (model present, but benchmark labels + split + training code absent). This is not evidence of fabrication — the benchmark simply was not deposited — but it means the paper's central performance claims cannot be verified from what was shared. Flagged per HARD RULE 5.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
The shipped pretrained model + inference pipeline are genuine: in the paper's exact env, main.py on test.fasta reproduces the demo outputs byte-for-byte (C1, grade exact). However, the paper's central performance claims — Table 1 independent-test (ACC 83.14%, MCC 0.663, AUC 0.907) and 5-fold CV (ACC 84.03%, MCC 0.681, AUC 0.913) — are not independently derivable from what was deposited: the labeled benchmark, the train/test split, and all training code are absent (Zenodo = repo zip). This is an authors'/data-availability defect, not a measured discrepancy — there is no evidence of fabrication, but the headline numbers cannot be verified from the shared artifacts, which is the critical gap here.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.