Predicting the pathogenicity of missense variants using features derived from AlphaFold2.
The main results reproduced: recomputed values matched the published ones within tolerance.
- Nothing in this column.
- 🟡Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
REPRODUCED (1:1, within-tol). AlphScore = AlphaFold2-feature random-forest missense pathogenicity predictor (Bioinformatics 2023). Described well enough: the Snakemake repo (github.com/Ax-Sch/AlphScore) is complete and the deposited genome-wide score table (Zenodo 6288139, 66,931,527 variants = reported N exactly) ships AlphScore + CADD/REVEL/DEOGEN2 + the final combined glm scores + gnomAD/ClinVar flags. Strategy: re-derive the headline numbers directly from the deposited product, mirroring get_performance_clinvar_2022.R (ClinVar 20220109 ROC-AUC, C1-C7) and compile_scores.R+analyse_DMS_score.R (13 DMS experiments mean |Spearman|, C8-C9). All 9 targeted claims reproduce within ~0.01-0.02: AlphScore ClinVar AUC 0.7975 vs 0.799 (Delta 0.002) and DMS mean Spearman 0.3525 vs 0.344 (Delta 0.009). NOT attempted: from-scratch retrain (RF + AlphaFold2 feature extraction) - the development variant sets and models are not deposited and the FEATURE framework is access-gated; and the exact held-out protein split (not deposited). No fabrication indicators - every reported value is derivable from shipped data + public ClinVar/DMS. Verdict provisional; human reviewer signs off.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-19 ⛓ f8a734b1ac53
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-30
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetWhether AlphaFold2-predicted wild-type protein structures can be used to derive features that improve computational (in silico) prediction of the pathogenicity of missense variants.
- ★ AlphaFold2-derived structural features (solvent accessibility, amino acid network features, physicochemical environment, pLDDT) can be used to train a random forest classifier (AlphScore) that distinguishes proxy-benign from proxy-pathogenic missense variants. method
- ★ AlphScore alone performs worse than established in silico scores such as CADD and REVEL. finding
- ★ Combining AlphScore with existing scores (CADD, REVEL, DEOGEN2) increases predictive performance for missense pathogenicity, as measured by DMS data approximation and ClinVar classification. finding
- ★ AlphaFold2-derived features are a genuine, important contributor to variant classification, since removing them (NullModel) substantially reduces performance. finding
- Pathogenic/proxy-pathogenic variants tend to occur in protein regions with higher pLDDT (AlphaFold2 confidence) values. finding
- AlphScore, its combinations with existing scores, and the training/testing datasets are made publicly available. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| machine learning classification (random forest, gradient boosting, extremely randomized trees) | gnomAD v3.1 and ClinVar missense variants across the human proteome | none | AUROC distinguishing proxy-benign/benign vs proxy-pathogenic/pathogenic variants | R packages ranger and xgboost |
| secondary structure and solvent-accessible surface area extraction | AlphaFold2-predicted structures of the human proteome | none | secondary structure and solvent accessibility per residue | DSSP v3.0.0 |
| physicochemical environment feature extraction | AlphaFold2-predicted structures | none | physicochemical properties within 0, 3, and 6 Å spheres around C-alpha atoms | FEATURE framework v3.1.0 |
| amino acid network construction and analysis | AlphaFold2-predicted structures | none | network-based metrics and weighted neighbor-residue properties | biographs and networkx (4 Å atom-atom cutoff) |
| permutation-based feature importance | AlphScore random forest model (218 features) | feature value permutation | ranked importance of predictive features | R package ranger |
| deep mutational scan (DMS) correlation analysis | 10 proteins: UBE2I, SUMO1, TPK1, BRCA1, P53, HRAS, PTEN, ADRB2, MSH2, VKOR1 | comprehensive missense mutagenesis (published DMS datasets) | Spearman correlation between DMS functional scores and computational prediction scores | — |
| AUROC classification performance evaluation | ClinVar_test missense variants (benign/likely benign vs pathogenic/likely pathogenic) | none | AUROC for AlphScore, CADD, REVEL, DEOGEN2 and their combinations | — |
| subgroup AUROC comparison by protein class | ClinVar_test variants split into membrane-associated vs non-membrane-associated proteins | none | AUROC per protein subgroup | PANTHER database v17.0 |
- – AlphScore (random forest) achieved the best performance among tested ML algorithms on the validation set AUROC=0.793
- ▼ NullModel, lacking AlphaFold2-derived features, performed substantially worse than AlphScore AUROC=0.609
- ▲ CADD outperformed AlphScore alone on the same validation set AUROC=0.871
- ▼ Removing pLDDT-based features from AlphScore caused only a marginal performance decrease AUROC 0.793 to 0.790
- – Among the top 25 important features, solvent accessibility, amino acid network, physicochemical environment, and pLDDT feature categories dominated
- ▲ Proxy-pathogenic gnomAD and (likely) pathogenic ClinVar variants tend to localize to higher-pLDDT protein regions
- other AUROC=0.793 (AlphScore performance on ClinVar_val)
- other AUROC=0.609 (NullModel (no AlphaFold2 features) performance on ClinVar_val)
- other AUROC=0.871 (CADD performance on ClinVar_val)
- other AUROC=0.790 (AlphScore performance on ClinVar_val after removing pLDDT features)
- count 315876 (gnomAD_train variant set size)
- count 21068 (ClinVar_test total variants)
- count 218 (number of structural features extracted per amino acid)
- count 92% (190258/205726) (proportion of ClinVar missense variants classified as VUS or conflicting interpretation)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The study trained tree-based machine-learning classifiers (gradient boosting, random forest, extremely randomized trees) to distinguish proxy-benign from proxy-pathogenic missense variants derived from gnomAD, selecting the best model (AlphScore, a random forest) via a grid search evaluated by AUROC on an independent ClinVar validation set. Performance was then assessed on a held-out ClinVar test set and on deep mutational scanning (DMS) data using Spearman correlation, and AlphScore was combined with existing scores (CADD, REVEL, DEOGEN2) via logistic regression. Uncertainty in AUROC and rank-score statistics was estimated by non-parametric bootstrapping, and permutation-based feature importance was used to interpret the model.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| AUROC comparison | AlphScore vs NullModel vs CADD/REVEL/DEOGEN2 and combined scores on ClinVar_val and ClinVar_test | ClinVar_val n=35,640; ClinVar_test n=21,068 | not stated |
| Spearman correlation (cor.test in R) | computational scores vs DMS data across 13 protein datasets; also correlation between a modified training-filter model and AlphScore on independent ClinVar variants | varies per DMS dataset (datasets with <20 variants for any score were excluded) | not stated |
| Logistic regression (R glm, family=binomial(link="logit")) | combining AlphScore with CADD, REVEL, and DEOGEN2 into combined prediction scores | ClinVar_val, n=35,640 | not stated |
| Permutation-based feature importance (ranger, importance="permutation") | ranking the 218 features used by AlphScore | gnomAD_train, n=315,876 | not stated |
| Non-parametric bootstrapping (R package boot, 1000 resamples) | uncertainty estimation for AUROC (ClinVar_val/test) and rank-score statistics (DMS data) | 1000 bootstrap samples | na |
| Grid search model comparison (gradient boosting, random forest, extremely randomized trees) via AUROC | selecting the best-performing algorithm/hyperparameters (AlphScore) | ClinVar_val, n=35,640 | not stated |
-
Model performance was primarily summarized from a single train/validation/test split of the data, partitioned by protein (80%/20%).↳ Could also: k-fold cross-validation or repeated random splits across proteins — This would also yield a distribution of AUROC estimates across folds, showing how sensitive performance is to the particular split chosen, complementing the single-split estimate already reported.
-
Uncertainty in AUROC and rank-score statistics was estimated using non-parametric bootstrapping.↳ Could also: Analytic approaches such as DeLong's method for AUROC standard errors and comparisons — This would also enable a formal statistical test comparing AUROCs between AlphScore, CADD, REVEL, and combined scores, alongside the descriptive bootstrap-based uncertainty already provided.
-
Multiple prediction scores were compared across 13 DMS datasets and multiple ClinVar-based analyses without a stated adjustment for multiple comparisons.↳ Could also: A correction such as Benjamini-Hochberg FDR or Bonferroni applied across the family of correlation/AUROC comparisons — This would also help control the overall false-positive rate when many related comparisons of scores and datasets are considered together.
-
Scores were combined using standard logistic regression (glm, binomial/logit link).↳ Could also: Regularized regression (e.g., elastic net) or a nonlinear ensemble learner (e.g., gradient boosting) for combining scores — This would also accommodate potential nonlinear interactions between component scores, and could be evaluated within the same train/validation framework already used.
-
Correlation between computational scores and DMS data was calculated per protein/dataset with Spearman correlation, then summarized as a mean absolute correlation across 13 datasets.↳ Could also: A mixed-effects or meta-analytic model treating protein/dataset as a random effect — This would also formally account for between-protein heterogeneity in the correlation estimates rather than combining them as a simple unweighted average.
-
Feature importance was assessed using permutation-based importance from the fitted random forest model.↳ Could also: SHAP (Shapley additive explanations) values — This would also provide per-variant, direction-aware attributions of feature contributions, complementing the global permutation-importance ranking already reported.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-37084271 (AlphScore)
Paper: Schmidt A, Röner S, Mai K, Klinkhammer H, Kircher M, Ludwig KU.
"Predicting the pathogenicity of missense variants using features derived from
AlphaFold2." Bioinformatics 2023. PMID 37084271 · DOI 10.1093/bioinformatics/btad280.
Code: https://github.com/Ax-Sch/AlphScore (GPL-3.0, last push 2023-08-25, NOT archived).
Data: Zenodo 10.5281/zenodo.6288139 (AlphScore_final.tsv.gz, 9.38 GB + .tbi).
What the paper does (computational pipeline)
A Snakemake pipeline:
- Download all human AlphaFold2 structures (
UP000005640_9606_HUMAN_v2.tar, EBI). - Extract ~218 per-residue structural features (DSSP, FEATURE framework v3.1.0, protinter, biopython HSExposure, biopandas pLDDT, networkx graph features).
- Merge features into dbNSFP 4.2a; form variant sets:
- gnomAD training set: proxy-benign (AF > 0.1%) vs proxy-pathogenic (NFE singletons). Paper: n = 315,876 variants.
- ClinVar validation/test sets (split by protein: 80% train-proteins → val, 20% held-out proteins → test). Paper: ClinVar_val n=35,640; ClinVar_test n=21,068 (9,224 benign + 11,844 pathogenic).
- Train a random forest (R
ranger, num.trees=2000, max.depth=5, min.node.size=10) → AlphScore. - Combine AlphScore with existing scores (CADD, REVEL, DEOGEN2) via logistic- regression glm fitted on an interim ClinVar set.
- Evaluate: ClinVar ROC-AUC (
get_performance_clinvar_2022.R) and DMS Spearman correlation (analyse_DMS_score.R) across 13 deep-mutational- scanning experiments in 11 proteins.
Reported results targeted (claims.tsv)
| id | what | reported |
|---|---|---|
| C1 | AlphScore alone ROC-AUC, ClinVar test | 0.799 (val 0.793) |
| C2 | CADD alone ROC-AUC, ClinVar test | 0.885 |
| C3 | REVEL alone ROC-AUC, ClinVar test | 0.929 |
| C4 | DEOGEN2 alone ROC-AUC, ClinVar test | 0.886 |
| C5 | AlphScore+CADD ROC-AUC | 0.909 |
| C6 | AlphScore+REVEL ROC-AUC | 0.935 |
| C7 | AlphScore+DEOGEN2 ROC-AUC | 0.907 |
| C8 | AlphScore mean | Spearman |
| C9 | CADD→+AlphScore mean | Spearman |
| C10 | DEOGEN2→+AlphScore | 0.422 → 0.436 |
| C11 | REVEL→+AlphScore | 0.421 → 0.442 |
| C12 | AlphScore+DEOGEN2+REVEL (best combo) | 0.450 |
| C13 | AlphScore_final computed for n variants | 66,931,527 (80% of dbNSFP) |
In scope vs out of scope
- In scope (pipeline-derived): all of C1–C13 are computational outputs of the AlphScore pipeline / its R evaluation scripts.
- Out of scope (external/wet-lab): the DMS fitness measurements themselves
(published by other labs; shipped in repo
resources/as supplementary xlsx) — used as a fixed external validation target, not reproduced.
Reproduction strategy + honest constraints
The headline numbers depend on pipeline-internal intermediate files that are
NOT deposited — the only Zenodo artifact is the final score table
AlphScore_final.tsv.gz. Specifically get_performance_clinvar_2022.R and
analyse_DMS_score.R both require pre_final_model_regular_variants.csv.gz
(carries the protein-level train/val/test split flags + per-variant predicted_Alph
cross-validated AlphScore) and, for DMS, validation_set_w_AlphScore.csv.gz. Those
are generated only by the full pipeline.
Full-pipeline regeneration (testing=False) is impractical / partly blocked:
- FEATURE framework v3.1.0 is gated behind a simtk.org registration wall (manual account) → soft blocker for the 218-feature extraction.
- dbNSFP 4.2a (~30 GB, academic FTP
dbnsfp:dbnsfp) + all human AlphaFold structures. - >100,000 SLURM jobs, ~500 GB scratch — would monopolise the shared cluster.
testing=Truemode (~213 proteins) runs end-to-end but trains on a tiny subset → produces the right kind of output, not the paper's numbers (executability only).
Chosen path — independent re-derivation from deposited data (valid per brief P16): Apply the analysis to the paper's
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.