Evaluating native-like structures of RNA-protein complexes through the deep learning method.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- Nothing in this column.
- 🟡Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
DRPScore (Nat Commun 2023) = a 4D-CNN that scores RNA-protein docking decoys. We cloned the authors' own repo (@8114812) + 260MB Git-LFS data to «our HPC» «infra», built a PyTorch env, and ran the shipped pipeline (Main.py voxelization -> pretrained 4DCNN.pth -> pred.py softmax) on all 13 Detail_Data(figure5) complexes (13000 decoys) on «our HPC» compute nodes. RESULT: the reproduced per-decoy native-probabilities are IDENTICAL to the authors' own published scores (max|diff|<=1e-6, Pearson=1.000) for 12/13 complexes (1SER partial, 745/1000 decoys), and our independent Top-N(iRMS<=4A) success classification matches the authors' for 13/13 complexes at every N -> the scoring pipeline reproduces essentially bit-for-bit. The 13-complex (unbound) aggregate is Top-5 46.15% / Top-20 61.54%, consistent with the paper's reported unbound rates (43.86/56.14 over 57 complexes). HOWEVER the paper's HEADLINE Top-N percentages (bound-bound 80.56/91.67 over 36 complexes; unbound over the full 57) cannot be regenerated from the deposit because those decoy ensembles were never deposited (testing_set1.1/1.2 ship only one native PDB per complex). Honest verdict: the method/pipeline is reproduced EXACTLY on all data that was deposited; coverage of the headline benchmark numbers is partial due to missing decoy data. No sign of fabrication in the deposited figure-5 data (our independent rerun matches it). NOT attempted: from-scratch retraining, external 3dRPC decoy generation.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-30
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-30no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetA deep-learning-based scoring function (DRPScore) can more accurately identify native-like RNA-protein complex structures than existing statistical potential-based scoring functions, including in the more challenging case of binding-induced conformational flexibility (unbound docking).
- ★ DRPScore identifies native-like RNA-protein structures with higher success rates than ITScore-PR, DARS-RNP, and 3dRPC across bound and unbound testing sets. finding
- ★ DRPScore's 4D convolutional neural network captures both local (intra-nucleotide/residue) and global (inter-nucleotide/residue, including secondary structure) features, unlike traditional 3DCNN. method
- ★ DRPScore consistently outperforms existing methods even on the most difficult fully flexible unbound-unbound docking cases. finding
- ★ DRPScore significantly improves accuracy of native interface interaction (hydrogen bond) predictions compared to existing scoring functions. finding
- DRPScore models the RNA-protein interaction interface within a 6 Å cutoff using 85 RNA atom types and 225 protein atom types with mass and charge fed into a grid-based CNN. method
- Training decoys for DRPScore were generated using physics-based simulations rather than relying only on native structures. method
- The traditional 3DCNN model, lacking inter-residue connections, is unable to accurately identify native-like RNA-protein complexes. finding
- DRPScore is proposed as a broadly useful resource for modeling and designing RNA-protein complexes. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Structure scoring/ranking benchmark (RMSD-based success rate) | Bound-bound RNA-protein complexes (three generated testing sets) | none (rigid-body docking decoys) | Success rate of top N (5-1000) predictions identifying native-like structure (Irmsd) | DRPScore (4DCNN) vs ITScore-PR, DARS-RNP, 3dRPC |
| Structure scoring/ranking benchmark (RMSD-based success rate) | Unbound-bound RNA-protein complexes (testing set from Huang and Zou) | conformational change upon binding (unbound docking) | Success rate of top N predictions, lowest RMSD per complex | DRPScore (4DCNN) vs ITScore-PR, DARS-RNP, 3dRPC |
| Structure scoring/ranking benchmark (RMSD-based success rate) | Unbound-unbound RNA-protein complexes (testing set II) | fully flexible docking (both partners unbound) | Success rate of top 10/20/30/40 predictions | DRPScore (4DCNN) vs ITScore-PR, DARS-RNP, 3dRPC |
| Interface hydrogen bond analysis | Bacillus subtilis YxiN protein-23S rRNA fragment complex (PDB 3MOJ) | none (comparison of lowest-RMSD predicted structures per method) | Number of interface hydrogen bonds and percent agreement with native structure | HBPULS |
| Secondary structure prediction/analysis | Protein and RNA components of RNA-protein interface (e.g., 3MOJ) | none | Fraction of hydrogen bonds corresponding to loop-helix secondary structure interactions | PSIPRED (protein), forna (RNA) |
| Model architecture comparison (structure scoring benchmark) | Bound-bound RNA-protein testing sets | none | Average success rate and standard deviation of native-like structure identification | DRPScore/4DCNN vs traditional 3DCNN |
| Robustness/redundancy analysis (structure scoring benchmark) | Testing set II at full and 0.8 sequence redundancy cutoff | sequence redundancy filtering | RMSD and success rate performance across redundancy levels | DRPScore vs ITScore-PR, DARS-RNP, 3dRPC |
- ▲ Bound-bound top 5 predictions: DRPScore success rate higher than all comparators 80.56% vs 79.63%/70.37%/64.81%
- ▲ Bound-bound top 20 predictions: DRPScore success rate highest 91.67% vs 89.81%/86.11%/83.33%
- ▲ Unbound-bound/unbound-unbound top 5 predictions: DRPScore success rate highest 43.86% vs 38.60%/35.09%/36.84%
- ▲ Unbound testing top 20 predictions: DRPScore success rate highest, ~10.53-15.79% improvement over others 56.14% vs 45.61%/40.35%/42.11%
- ▼ 3MOJ example: DRPScore top-5 lowest RMSD structure far closer to native than other methods Irmsd = 1.98 Å vs 14.74/8.28/8.44 Å
- ▲ 3MOJ interface hydrogen bonds identified by DRPScore agree substantially more with native structure than other methods 62.5% agreement (8 H-bonds) vs 0% for ITScore-PR/DARS-RNP/3dRPC
- ▲ Unbound-unbound (set II) top 20 average success rate highest for DRPScore 58.5% vs 51.2%/48.8%/46.3%
- ▼ 1JID example: DRPScore lowest RMSD model in top 50 predictions closer to native than other methods 1.92 Å vs 6.58/2.89/4.82 Å
- count 20 predictions with RMSD <4 Å for DRPScore vs 14 (ITScore-PR), 9 (DARS-RNP), 14 (3dRPC) (unbound-bound testing set interface ranking analysis)
- count 8 predictions with RMSD >8 Å for DRPScore vs 15 (ITScore-PR), 17 (DARS-RNP), 14 (3dRPC) (unbound-bound testing set interface ranking analysis)
- mean DRPScore average RMSD 8.94 Å (3LRR) vs ~10.0 Å for other methods (top 50 predictions, unbound-bound example PDB 3LRR)
- mean DRPScore average RMSD 8.85 Å (1JID) vs ~11.0 Å for other methods (top 50 predictions, unbound-unbound example PDB 1JID)
- fold_change roughly 10.53-15.79% improvement over existing methods (unbound complex success rate improvement, discussed generally)
- other training: learning rate 0.0001, 20000 training steps, ~6 s/iteration, 6.6 GB memory, 12500-step model selected (4DCNN training on GeForce RTX 3070)
- other DRPScore evaluates 1000 RNA-protein complex structures in about 8 minutes (computational runtime)
- other 3 independent tests used for bound-bound success rate mean/SD (Fig. 1 bound-bound testing set statistics)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a computational/methods paper introducing DRPScore, a deep-learning scoring function for identifying native-like RNA-protein complex structures, benchmarked against three existing statistical potential scoring functions (ITScore-PR, DARS-RNP, 3dRPC) and a traditional 3DCNN baseline. Performance was assessed primarily via success rates (percentage of top-N predictions with RMSD below native-structure thresholds) across bound-bound, unbound-bound, and unbound-unbound RNA-protein docking test sets, along with RMSD distributions and case-study comparisons for individual complexes. The only explicit quantitative variability reporting is mean success rate ± standard deviation across 3 independent test sets for the bound-bound benchmark; no formal hypothesis tests, p-values, or multiplicity corrections are described anywhere in the text.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Descriptive summary (mean success rate ± standard deviation) across independent test sets; no formal inferential/hypothesis test named | Fig. 1 — bound-bound RNA-protein testing sets, comparing DRPScore, ITScore-PR, DARS-RNP, 3dRPC | number = 3 independent tests (as stated in Fig. 1 legend) | not stated |
-
Differences in success rates between DRPScore and the comparator methods (ITScore-PR, DARS-RNP, 3dRPC) are presented as point percentages without an accompanying inferential test.↳ Could also: A paired test on a per-complex basis (e.g., McNemar's test for paired binary success/failure outcomes, or a paired Wilcoxon signed-rank test on per-complex RMSD ranks) — Since each method is evaluated on the same set of complexes, a paired approach would use that matched structure and could quantify whether the observed performance differences exceed what might be expected from sampling variability.
-
Variability in the bound-bound benchmark is summarized as mean ± SD across 3 independent tests.↳ Could also: A 95% confidence interval (e.g., via bootstrap resampling of the top-N outcomes) or a Wilson score interval for the underlying success proportion — With an n of 3, a CI can more directly communicate the precision of the estimated success rate than SD alone, and bootstrap or exact-proportion intervals are commonly used for small-sample success-rate estimates.
-
Success rates are compared across several scoring functions, several test-set types (bound-bound, unbound-bound, unbound-unbound), and several top-N thresholds within the same set of complexes.↳ Could also: A multiplicity-adjustment method such as Bonferroni or Benjamini-Hochberg FDR applied across the family of pairwise method comparisons — When many comparisons are drawn from the same underlying data, an adjustment procedure helps control the overall false-positive rate across the full set of comparisons.
-
Deep-learning model performance (DRPScore, 3DCNN) is reported from a single trained model (e.g., the 12500-step checkpoint) rather than across repeated training runs.↳ Could also: Reporting variance across multiple training runs with different random seeds, or k-fold cross-validation on the training decoys — Repeated-run or cross-validation summaries can characterize how much of the observed performance is attributable to training stochasticity versus the modeling approach itself.
-
Shifts in RMSD distributions toward lower values for DRPScore are described qualitatively from histograms (Fig. 4) rather than through a distributional comparison statistic.↳ Could also: A two-sample distributional test such as the Kolmogorov-Smirnov test or Mann-Whitney U test comparing RMSD distributions between methods — These tests could provide a formal, quantitative complement to the visual histogram comparison of distribution shifts.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-36828844 (DRPScore)
Paper: Zeng et al. 2023, Nat Commun 14:1060. "Evaluating native-like structures of RNA-protein complexes through the deep learning method." DOI 10.1038/s41467-023-36720-9 · PMCID PMC9958188.
Method (DRPScore): a 4D convolutional neural network (4DCNN) that scores RNA-protein docking decoys. Input = tensor 1×3×L×(32×32×32) (features: occupation/mass/charge of atoms in voxel grid at the RNA-protein interface within 6 Å; L=128 max RNA length). Output = probability that a structure is native-like. Decoys are ranked by this probability; Top-N success = at least one near-native (interface RMSD ≤ 4 Å, CAPRI criterion) appears in the top N.
Pipeline used (this is the authors' own code, P16 N/A — own repo)
GitHub Zhaolab-GitHub/DRPScore_v1.0 @ commit 81148123421ef63650a23c9d99dcd471a395ddfa.
Three-step shipped pipeline:
6A_calculation/rpo.py— extract 6 Å RNA-protein interface atoms from a PDB.data/predict/Main.py— build the*.npyscoring-prep tensor.pred.py— 4DCNN inference →scoring.txt(col3 = native probability). Pretrained 4DCNN weights ship insideDRPScore_4DCNN.zip. Test decoys + RMSD labels ship insidetraining_testing_set.zip.
IN SCOPE (pipeline-derived, attempt to reproduce)
Top-N decoy-ranking success rates produced by running DRPScore on the shipped test decoys:
| ID | Result | Reported | Paper loc |
|---|---|---|---|
| C1 | Bound-bound (36 cplx) Top-5 success | 80.56% | Fig.1 / text |
| C2 | Bound-bound (36 cplx) Top-20 success | 91.67% | Fig.1 / text |
| C3 | Unbound (57 cplx) Top-5 success | 43.86% | Fig.2 / text |
| C4 | Unbound (57 cplx) Top-20 success | 56.14% | Fig.2 / text |
| C5 | Unbound-unbound (41 cplx) Top-20 avg | 58.5% | Suppl Fig.8 |
Primary 80/20 target = C1/C2 (bound-bound: 36 complexes, fastest, cleanest). Then push to C3/C4 (unbound, larger/harder), then C5.
OUT OF SCOPE (not pipeline-reproducible here)
- 4DCNN from-scratch retraining on the 277-complex training set (501 structures each ≈ 139k; would need decoy regeneration via 3dRPC + multi-GPU training). We use the shipped pretrained weights for inference — this is the faithful evaluation pipeline; retraining is a separate, much larger effort.
- Decoy generation with 3dRPC / RPDOCK (external docking tool, not in repo). We use the shipped decoys.
- Comparator scores (ITScore-PR, DARS-RNP, 3dRPC) — external scoring functions, not part of this repo; their reported numbers are taken as-is, not regenerated.
- Wet-lab / structural-biology claims (H-bond interface analysis of 3MOJ etc.) — manual structural interpretation, not a pipeline output.
Datasets the paper relies on (profiled in same pass)
- GitHub LFS (the REAL data):
DRPScore_4DCNN.zip126 MB (model+code),training_testing_set.zip35.6 MB (train+test decoys+labels),Detail_Data(figure5).zip102 MB. - Zenodo 10.5281/zenodo.7614606: only a 3.1 KB GitHub auto-snapshot that contains LFS pointer files, NOT the data → does NOT deliver the data on its own (profiling note).
- External refs (not deposits): NDB database, Zoulab RNA docking benchmark.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
The DRPScore scoring pipeline reproduces the authors' own published per-decoy scores essentially bit-for-bit (max|diff|<=1e-6, Pearson=1.000 for 12/13 figure-5 complexes; Top-N classification 13/13), and the 13-complex unbound subset (46.15%/61.54%) is consistent with the reported 43.86%/56.14% — no fabrication signal. The deviation is one of coverage, not correctness: the paper's headline benchmark numbers (C1/C2 36-complex 80.56%/91.67%, full 57-complex C3/C4) cannot be regenerated because those decoy ensembles were never deposited (deposit graded D, delivers_promised=no). That gap lies on the authors' side (q4 red) as a data-availability defect, not our methodology, and the subset reproduction redefines the cohort (13 vs 57). Overall a strong, explainable-deviation reproduction → yellow.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.