Comprehensive benchmarking of large language models for RNA secondary structure prediction.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No relevant deviation in data/preprocessing
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- ✓Overall, the reproduction was clean
- Every checked point held up.
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
REPRODUCED (within tolerance). Re-ran the deposited pipeline (frozen precomputed RNA-LLM embeddings -> trainable ResNet head) on ArchiveII random 5-fold CV at the paper config (lr=1e-4, 15 epochs) for ALL 7 feature sets. Every model matches the authors' deposited ArchiveII F1 within |dF1|<=0.0089 (per-fold mean for the 5 models with deposited test metrics; per-seq median vs scores.csv for rnafm+RiNALMo, whose per-fold test metric was not deposited -> we fill the gap). Model ranking identical: one-hot < RNABERT < RNA-MSM < RNAErnie < RNA-FM < {ERNIE-RNA, RiNALMo}. Max |delta| over 14 comparisons = 0.0089; no seed in code so this is essentially exact reproduction within run-to-run noise. KEY METHOD: the 4-node H100 partition was indefinitely booked, but embeddings are precomputed so the only compute is the tiny head over <=510nt contact maps -> reproduced entirely on CPU (std partition), 35-task array done in ~2.3h. Config provenance: lr=1e-4 confirmed by viewing the PMC equation image (=repo default, no discrepancy). FLAG for reviewer: paper text 'top-3 ERNIE-RNA/RiNALMo 0.85, RNA-FM 0.82' sit nearest the deposited per-SEQ MEANS, not the per-seq medians (.954/.954/.913); deposited scores.csv is authoritative. NOT attempted: famfold, bpRNA/bpRNA-new, PDB-RNA, classical baselines.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 88assessed: 2026-06-20 ⛓ 08c28ab49434
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-20
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-20no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper tests whether representations learned by pretrained RNA large language models (RNA-LLMs) can enhance the data-costly task of RNA secondary structure prediction, and whether existing RNA-LLMs generalize well to new, low-homology structures when evaluated in a single unified experimental setup.
- ★ Existing RNA-LLMs had not previously been evaluated for secondary structure prediction in a unified, fair experimental setup with the same datasets and prediction model. finding
- ★ Two of the evaluated RNA-LLMs clearly outperform the other models on secondary structure prediction. finding
- ★ Generalization to low-homology scenarios (unseen RNA families) is significantly challenging for all evaluated RNA-LLMs. finding
- ★ The study provides curated benchmark datasets of increasing complexity (random split, cross-family/leave-one-family-out) for evaluating RNA-LLM representations on secondary structure prediction. resource
- ★ A common deep learning architecture was used downstream of each RNA-LLM's embeddings to enable fair comparison across models. method
- A single-sequence version of RNA-MSM was used (instead of its native MSA-based input) to ensure a fair comparison with the other single-sequence RNA-LLMs. method
- Source code and benchmark datasets are publicly released at https://github.com/sinc-lab/rna-llm-folding/. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| RNA secondary structure prediction using RNA-LLM embeddings + common deep learning architecture | ArchiveII benchmark dataset (9 RNA families: 5s, srp, tRNA, tmRNA, RNaseP, grp1, 16s, telomerase, 23s), 3864 sequences | embedding source varied across six pretrained RNA-LLMs (RNABERT, RNA-FM, RNA-MSM, ERNIE-RNA, RNAErnie, RiNALMo) | secondary structure prediction performance, five-fold random partitions | — |
| Cross-family generalization test (leave-one-family-out) | ArchiveII dataset, trained on all families but one held-out test family, repeated for all families | embedding source varied across six pretrained RNA-LLMs | secondary structure prediction performance on unseen RNA family | — |
| RNA secondary structure prediction using RNA-LLM embeddings + common deep learning architecture | bpRNA dataset (nonredundant at 80% sequence identity, 13 419 RNAs; TR0 training set of 10 814 RNAs, as used in SPOT-RNA) | embedding source varied across six pretrained RNA-LLMs | secondary structure prediction performance | — |
| Self-supervised masked language model (MLM) pretraining | Large-scale ncRNA sequence corpora (RNAcentral, Rfam, Ensembl, nt, depending on model) | none (self-supervised pretraining of each RNA-LLM) | learned contextual base/token embeddings | — |
- – Two RNA-LLMs clearly outperform the other evaluated models on secondary structure prediction
- ▼ All models show significant challenges generalizing to low-homology (unseen family) scenarios
- count 3864 (Total number of sequences in the ArchiveII benchmark dataset)
- count 13 419 (Total nonredundant RNAs in the bpRNA dataset)
- count 10 814 (Number of RNAs in the bpRNA training set (TR0))
- count 512 nt (Maximum sequence length filter applied across all datasets)
- count 76 237 (Pretraining sequences used for RNABERT (human-derived small ncRNAs from RNAcentral))
- count 23 700 000 (Pretraining sequences used for RNA-FM (unannotated ncRNAs from RNAcentral))
- count 36 000 000 (Pretraining sequences used for RiNALMo (largest RNA-LLM, 650M parameters))
- count 3087 138 (Pretraining sequences (from 3932 Rfam families) used for RNA-MSM)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper presents a comparative benchmarking study of several pretrained RNA large language models (RNA-LLMs) as input representations for a common downstream secondary-structure prediction architecture. Evaluation uses dataset splits rather than classical inferential statistics as described in the provided excerpt: five-fold cross-validation on the ArchiveII dataset, a fixed train/test split (bpRNA, following the SPOT-RNA partition), and a leave-one-family-out design to assess cross-family generalization. The excerpted text does not include a Results/statistical-analysis section describing significance testing, dispersion measures, or multiplicity correction.
-
Model comparison across ArchiveII five-fold splits and the leave-one-family-out design appears to rely on point performance metrics per fold/family without a stated significance test between RNA-LLMs.↳ Could also: A paired test across folds/families (e.g., Wilcoxon signed-rank test) or a repeated-measures ANOVA treating fold/family as the repeated unit — This would let the comparison quantify whether observed performance differences between RNA-LLMs exceed what could be expected from fold-to-fold or family-to-family variability alone.
-
Six RNA-LLMs are evaluated and presumably ranked against one another across several benchmark datasets.↳ Could also: A multiple-comparison correction (e.g., Benjamini-Hochberg FDR or Bonferroni) applied to any pairwise model comparisons — Controls the family-wise or false-discovery rate when many pairwise comparisons among models and datasets are examined simultaneously.
-
Cross-validation (five-fold) and leave-one-family-out generalization are used as the evaluation design, but variability across folds/families is not described with a dispersion measure in the excerpted text.↳ Could also: Reporting mean ± SD or a 95% confidence interval of the performance metric across folds/families — Conveys the stability of each RNA-LLM's performance across partitions, not just a single aggregated point estimate.
-
The leave-one-family-out strategy provides one generalization estimate per held-out family.↳ Could also: Repeated random subsampling or nested cross-validation with bootstrap confidence intervals — Produces a distribution of generalization estimates, which can make comparisons of low-homology generalization performance across models more robust to the particular family partitioning chosen.
-
Comparative claims (e.g., that two RNA-LLMs 'clearly outperform' the others) are made based on benchmark scores.↳ Could also: Bootstrap resampling of test-set predictions to generate confidence intervals around the metric differences between top-performing models — Allows readers to assess whether the gap between top models is likely to be a stable ranking or falls within the expected sampling variability of the test sets used.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — PMID 40205851
Title: Comprehensive benchmarking of large language models for RNA secondary structure prediction. DOI: 10.1093/bib/bbaf137 · PMCID: PMC11982019 Code: https://github.com/sinc-lab/rna-llm-folding (own code, public) Data: Zenodo 10.5281/zenodo.13821093 (precomputed embeddings, public, 21 files)
What this paper does (pipeline)
Benchmarks 6 pretrained RNA language models (RNABERT, RNA-FM, RNA-MSM, ERNIE-RNA,
RNAErnie, RiNALMo) + a one-hot baseline as frozen feature extractors. Per-nucleotide
embeddings feed a small trainable prediction head (SecondaryStructurePredictor:
FC → outer concatenation → 2× 2D-ResNet blocks → 2D conv) that predicts the base-pair
contact matrix. Trained with PyTorch 2.3, Adam, batch 4, 15 epochs, BCE loss
(repo default lr = 1e-4; Methods text states 1e-3 — discrepancy noted). Evaluated by
F1 / precision / recall of predicted base pairs.
Datasets of increasing difficulty: ArchiveII (random 5-fold CV + family-fold cross-family), bpRNA / bpRNA-new, PDB-RNA.
In scope (pipeline-derived, attempted)
- ArchiveII random 5-fold CV median F1 per embedding (Fig. 2a). Cheapest, fully
specified, low-hanging. Chosen targets: one-hot (~0.70), RNABERT (~0.82),
RNA-FM (~0.84). Entrypoint:
scripts/run_archiveii_kfold.py --emb <emb>_ArchiveII. Inputs ship in repo (data/ArchiveII.csv,data/ArchiveII_kfold_splits.csv); embeddings pulled from Zenodo as<emb>_ArchiveII.7z→data/embeddings/<emb>.h5.
In scope but not prioritized (the hard ~20%, deliberately deferred)
- ArchiveII family-fold (Fig. 3, per-family boxplots).
- bpRNA / bpRNA-new (embedding archives 1–11 GB each).
- PDB-RNA + classical baselines (LinearPartition-V etc.).
- ERNIE-RNA / RiNALMo ArchiveII (~0.88; 2–3.5 GB embeddings) — the headline best models, runnable but larger downloads; left as a fast follow-up.
Out of scope
- Pretraining of the 6 LLMs themselves (used as frozen, externally pretrained).
- Any wet-lab / manual / external-tool numbers (none material here).
Reference values are figure-read
The paper reports ArchiveII results as violin plots (Fig. 2a), not a numeric table. The target F1 medians (~0.70 … ~0.88) were read visually from the figure, so any reproduction comparison must use a loose tolerance and be treated as provisional. This is recorded in AUDIT.md.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
This is a clean, within-noise reproduction: all 7 ArchiveII random 5-fold CV F1 values match the authors' own Zenodo deposit within |ΔF1|≤0.0089 and the full model ranking is identical, so the central benchmarking claim holds. The only deviations are sub-0.01 deltas attributable to a seed-free pipeline (technical/expected, on neither our nor the authors' side as a defect). Minor deposit caveats — per-fold test metrics absent for RNA-FM and RiNALMo (gap-filled by our run) and 3864 vs canonical ~3975 sequences with undocumented filtering — are completeness gaps, not derivability or fabrication concerns. Overall: green across all 8 questions.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.