Enhanced Generalizability of RNA Secondary Structure Prediction via Convolutional Block Attention Network and Ensemble Learning.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- 🟡A deviation arose in the data or preprocessing
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
DESCRIBED WELL ENOUGH -> 1:1 reproduction. TrioFold (Molecules 2025) is a ~2800-param 9-channel CBAM ensemble head fusing the contact maps of 9 base learners. NOTE the brief's code_url (uci-cbcl/UFold) is WRONG: UFold is one base learner; the authors' own code is github.com/sfsdfd62/TrioFold @ fd379b8. Self-contained: repo ships 9-ch weights model/TrioFold.pt (sha d6d28057...), Zenodo 12714014 ships 3 pickles carrying precomputed base-learner outputs + ground truth. This room did a FRESH clean re-run («infra» workdir had been reclaimed by the janitor): one consolidated SLURM job (2218495, «our HPC» std/n112, CPU, 7m22s, exit 0) rebuilt the conda env, fetched the repo tarball, downloaded all 3 pickles (SHA256 of every file MATCHED the recorded values), pinned numpy 1.23.5, and ran the faithful CPU eval (run_eval.py mirrors TrioFold.py::model_eval_all_test exactly; only deviations device cuda:1->cpu and numpy pin, neither changes numerics). RESULTS: bpRNA-new Table 1 reproduces 1:1 to 3 decimals for TrioFold (F1 0.6593 vs 0.659; P 0.6136 vs 0.614; R 0.7360 vs 0.736) AND every base learner (UFold 0.608, MXfold2/EternaFold 0.633, CONTRAfold 0.639, ContextFold 0.580, SPOT-RNA 0.603, Mfold 0.601, RNAfold/LinearFold 0.617) -- all 12 exact (n=5388). Fig 4 medians: TS0 0.7170 vs 0.714 (within-tol, n=1287), TestSetA 0.8889 vs 0.909 (partial, ~2% low, n=558; reproduced median-precision is exactly 0.909 -- benign Fig-4/aggregation nuance flagged for the human, NOT fabrication). Numbers are bit-identical to the earlier 2026-06-16 run («job»), confirming determinism. DID NOT attempt: TrioFold-lite (4-channel weights not shipped) and PDB (Table 2)/TS-inter (Table 3) (those pickles not deposited). Re-running the 9 base learners from raw sequences and re-training were unnecessary (pickles carry their outputs; weights ship). No completeness claim; every comparison provisional and human-auditable. Integrity: 13/14 in-scope numbers match to 2-3 decimals; no fabrication signal. This re-run replaces the prior ROOM_RESULT that was requeued for missing the datasets[]/qc_room blocks (now present).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 87assessed: 2026-06-16 ⛓ 9f0cf25f848d
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-22
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-18no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper tests whether ensembling algorithms that learn diverse RNA folding principles (thermodynamic-based and deep-learning-based) can improve both prediction accuracy and generalizability of RNA secondary structure (RSS) prediction.
- ★ TrioFold integrates base-pairing clues from thermodynamic- and DL-based methods via ensemble learning and a convolutional block attention mechanism to enhance RSS prediction generalizability. method
- ★ TrioFold achieves higher accuracy on intra-family predictions and enhanced generalizability on inter-family and cross-RNA-type predictions. finding
- ★ RSS prediction algorithms show low concordance with each other despite comparable individual accuracies. finding
- ★ An ensemble of four base learners (SPOT-RNA + UFold + MXfold2 + ContextFold, TrioFold-lite) is optimal, since ensemble performance converges rather than increasing monotonically with more base learners. finding
- ★ TrioFold achieves the best performance among tested methods on unseen RNA family datasets (TS-inter, bpRNA-new, PDB-derived). finding
- ★ A free, user-friendly webserver (triofold.aiddlab.com) implementing TrioFold and base-learner algorithms plus analysis tools was developed for the RNA research community. resource
- TrioFold and TrioFold-lite have fewer trainable parameters than other compared algorithms. finding
- The ensemble model shows limited but non-zero ability to capture pseudoknotted base pairs. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Jaccard distance comparison of predicted contact maps | TestSetA RNA sequence dataset | none | pairwise Jaccard distance between algorithms' predicted structures and ground truth | — |
| PCA dimensionality reduction of prediction outputs | TestSetA RNA sequence dataset | none | clustering/separation of algorithm output similarity | — |
| Ensemble base-learner combination screening | TestSetA RNA sequence dataset | varying combinations/number of base learners (9 candidate models) | F1 score of ensemble model | — |
| Benchmark RSS prediction accuracy comparison (F1 score, ROC) | TestSetA and TS0 benchmark datasets | none | F1 score, ROC curve across TrioFold, TrioFold-lite and SOTA methods | — |
| RNA family/clan mapping and per-clan performance evaluation | bpRNA-new and other datasets | none | RNA family/clan distribution and prediction performance per clan | Infernal 1.1.4 |
| Sequence identity/data leakage check | TestSetA, TS0, bpRNA-new, PDB-derived datasets vs training set | none | percent sequence identity to training sequences | VSEARCH |
| Generalizability evaluation on unseen RNA families with paired t-test | TS-inter, bpRNA-new, PDB-derived unseen-family datasets | none | F1 score and per-sequence paired t-test p-value vs baseline algorithms | — |
| RSS structure visualization and alignment comparison | Individual RNA sequences from bpRNA-new (URS0000D6ACEB_12908_1-61, URS0000D69420_12908_1-94) and PDB structures (7KVT, 7U4A, 7EAF, 8I7N) | none | structure similarity score and F1 score vs ground truth | Forna; BEAGLE |
- – Jaccard distance between nine RSS prediction algorithms ranges 0.3–0.65 on TestSetA; EternaFold and CONTRAfold most concordant (0.31) 0.3-0.65
- ▲ Optimal ensemble size is four base learners; SPOT-RNA+UFold+MXfold2+ContextFold (TrioFold-lite) performs best
- ▲ TrioFold and TrioFold-lite achieve median F1 of 0.909 and 0.907 on TestSetA, 5.6% and 5.3% higher than the second-best model 5.6% and 5.3%
- ▲ On TS0, TrioFold and TrioFold-lite achieve median F1 of 0.714 and 0.733; TrioFold-lite improves 2.1%-38.6% over SOTA methods 2.1-38.6%
- ▲ TrioFold achieves best performance on all three unseen-family datasets (TS-inter, bpRNA-new, PDB)
- ▼ TrioFold and TrioFold-lite show lowest Jaccard distances to other algorithms (0.19-0.53 and 0.19-0.57; medians 0.427 and 0.414) 0.19-0.53
- – SPOT-RNA F1 score drops from 0.839 on TestSetA to 0.654 on TS0, showing inconsistent cross-dataset performance 0.839 vs 0.654
- ▲ TrioFold outperformed other methods on 3 of 4 newly released PDB structures (7KVT, 7U4A, 7EAF), second-best on 8I7N
- other Jaccard distance range 0.3-0.65 (concordance among 9 RSS algorithms on TestSetA)
- other Jaccard distance = 0.31 (EternaFold vs CONTRAfold, most concordant pair)
- mean median F1 0.909 (TrioFold) / 0.907 (TrioFold-lite) (TestSetA benchmark)
- fold_change 5.6% and 5.3% improvement over second-best model (TestSetA)
- mean median F1 0.714 (TrioFold) / 0.733 (TrioFold-lite) (TS0 dataset)
- fold_change 23.7%, 8.34%, 5.57%, 9.12% higher median F1 than UFold, SPOT-RNA, ContextFold, MXfold2 respectively (TestSetA vs individual base learners)
- count 96.27% (bpRNA-new) and 80.67% (PDB) of sequences have <30% identity to training set (data leakage check via VSEARCH)
- pvalue per-sequence paired t-tests (TrioFold vs each baseline algorithm on unseen family datasets)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a computational benchmarking study evaluating TrioFold, an ensemble deep learning model for RNA secondary structure (RSS) prediction. Performance across multiple benchmark datasets (TestSetA, TS0, TS-inter, bpRNA-new, PDB-derived) was quantified primarily by median F1 score and ROC curves. Per-sequence paired t-tests were used to assess statistical significance of improvements over each of nine baseline algorithms on the three unseen-family datasets. Inter-algorithm prediction concordance was characterized via pairwise Jaccard distances and PCA.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Per-sequence paired t-test | Comparing TrioFold against each baseline algorithm on TS-inter, bpRNA-new, and PDB-derived unseen-family datasets (Tables 1–3) | — | not stated |
| F1 score (harmonic mean of precision and recall) | Primary benchmark metric across all datasets (TestSetA, TS0, TS-inter, bpRNA-new, PDB); median reported | — | na |
| Jaccard distance (average pairwise) | Pairwise dissimilarity between contact-map predictions of all nine algorithms on TestSetA; extended to include TrioFold/TrioFold-lite (Figures 2A, S2, S3) | — | na |
| Principal Component Analysis (PCA) | Dimensionality reduction to visualize inter-algorithm prediction diversity across TestSetA sequences (Figure 2B) | — | na |
| ROC curve analysis | Performance visualization on TestSetA and TS0 (Figure 4C) | — | na |
-
Multiple per-sequence paired t-tests were conducted comparing TrioFold against each of nine baselines across three datasets, with no adjustment for multiple comparisons↳ Could also: Apply a false-discovery-rate procedure (e.g., Benjamini–Hochberg) or a family-wise correction (e.g., Bonferroni) across the family of pairwise tests — As the number of simultaneous tests grows, the probability of at least one spurious significant result increases; a correction procedure explicitly controls this rate and is standard practice in multi-baseline benchmarking studies
-
Paired t-tests on per-sequence F1 scores implicitly assume that the distribution of paired differences is approximately normal↳ Could also: Use the Wilcoxon signed-rank test as a non-parametric alternative — F1 score distributions can be bounded and skewed, particularly on small or structurally heterogeneous datasets; the Wilcoxon test requires no normality assumption for paired differences and is widely used in bioinformatics benchmarking
-
The primary performance summary is the median F1 score, reported as a point estimate with no measure of spread↳ Could also: Supplement with bootstrapped confidence intervals around the median F1 — A confidence interval conveys both the magnitude and the sampling uncertainty of the performance estimate, enabling readers to judge whether observed differences are likely stable across other sequence samples of similar composition
-
F1 score was the sole primary scalar metric for base-pair prediction accuracy↳ Could also: Also report Matthews Correlation Coefficient (MCC) or the area under the precision–recall curve (AUPRC) — Contact matrices are extremely sparse (far more non-paired than paired positions), so metrics that account for class imbalance—such as MCC or AUPRC—can complement F1 and are commonly reported alongside it in the RNA structure prediction literature
-
Inter-algorithm concordance was summarized as the average pairwise Jaccard distance across all test sequences↳ Could also: Report the distribution (e.g., IQR or per-sequence variance) of Jaccard distances alongside the mean, or use a multi-rater agreement statistic such as Krippendorff's alpha — Averaging collapses per-sequence variability; a spread measure or reliability coefficient would reveal whether high average concordance masks substantial disagreement on specific sequence subsets (e.g., pseudoknot-containing or long sequences)
-
Train/test independence was verified with a 30% sequence-identity threshold using VSEARCH↳ Could also: Supplement the identity cutoff with a structural-similarity-based or cluster-based partitioning check (e.g., using RNAforester scores as a similarity criterion) — Sequences below 30% sequence identity can still share conserved structural motifs; a complementary structure-based independence check would provide additional evidence that test performance is not inflated by cryptic similarity to training examples
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-40871599 (TrioFold)
Paper: Enhanced Generalizability of RNA Secondary Structure Prediction via Convolutional Block Attention Network and Ensemble Learning. Molecules 2025;30(16):3447. PMID 40871599 · PMCID PMC12388828 · DOI 10.3390/molecules30163447.
Method: TrioFold — a small (~2800-parameter) ensemble head that fuses the
outputs of 9 base learners (6 ML: MXfold2, UFold, SPOT-RNA, EternaFold, CONTRAfold,
ContextFold; 3 thermodynamic: Mfold, LinearFold, RNAfold) using a Convolutional
Block Attention Module (CBAM: channel "RSS Pattern Block" + spatial "Domain Focus
Block"). A TrioFold-lite variant also reported.
Corrected artifact pointers (brief enrichment was wrong)
- The brief's
code_url = github.com/uci-cbcl/UFoldis incorrect — UFold is only one of six base learners. The authors' own code is https://github.com/sfsdfd62/TrioFold (mirror: figshare 26377156). - Data: Zenodo 10.5281/zenodo.12714014 — three pickles (281 MB total):
bpRNAnew.cPickle(207 MB),TestSetA.cPickle(22.7 MB),TS0_repickled.cPickle(51.1 MB). These contain the precomputed base-learner outputs + ground-truth structures (the repo states it cannot ship the base learners "due to copyright issues", and the run instruction is simply: download pickles →python TrioFold.py). - Trained model weights ship in-repo:
model/TrioFold.pt(8.5 KB — consistent with the ~2800-parameter claim). So inference is fully self-contained.
In scope (pipeline-derived, reproducible)
TrioFold inference + F1/precision/recall evaluation, reproduced by running the
authors' TrioFold.py (loads a Zenodo pickle + model/TrioFold.pt, prints averaged
precision/recall/F1). One pipeline, one command per dataset:
| Result | Dataset (pickle) | Reported | Source |
|---|---|---|---|
| TrioFold F1 | bpRNA-new (bpRNAnew.cPickle) |
0.659 | Table 1 |
| TrioFold median F1 | TestSetA (TestSetA.cPickle) |
0.909 | Fig 4 / text |
| TrioFold median F1 | TS0 (TS0_repickled.cPickle) |
0.714 (lite 0.733) | Fig 4 / text |
Directly reproducible because the pickles carry base-learner outputs and the head
weights ship in-repo. Note: TrioFold.py prints the average F1; the paper
reports median F1 for TestSetA/TS0 (Fig 4), so those two are mean-vs-median and
must be graded as "comparable, metric-aggregation differs" unless the per-sample
F1 array is dumped and the median taken. bpRNA-new (Table 1) is the cleanest 1:1.
Out of scope (not attempted)
- Re-running the 9 base learners from raw sequences (repo explicitly does not ship them; copyright). Not needed — the pickles already contain their outputs.
- Re-training TrioFold from scratch (weights shipped; only inference is claimed-reproducible).
- PDB (Table 2) and TS-inter (Table 3) datasets: pickles not in the Zenodo record (only bpRNAnew/TestSetA/TS0 are), so those tables are not directly reproducible from the deposited data → out of scope.
- Statistical-significance annotations, webserver, figures.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
This is a strong, essentially 1:1 reproduction: bpRNA-new Table 1 reproduces to 3 decimals for TrioFold (F1 0.6593 vs 0.659) and all 9 base learners (n=5388), and the central claim that the CBAM ensemble beats every base learner holds exactly. The only blemishes are on the data-availability/figure side, not the authors' computation: the TestSetA Fig-4 median is ~2% low (0.8889 vs 0.909, a benign aggregation nuance — precision-median is exactly 0.909), and TrioFold-lite plus the PDB/TS-inter tables are not reproducible because those weights/pickles were never deposited. No fabrication signal; graded yellow overall only because it is not flawless across every reported number.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.