RNAmountAlign: Efficient software for local, global, semiglobal pairwise and multiple RNA sequence/structure alignment.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Reported values were directly comparable
- ✓Reported values are derivable from the shared data
- ✓The central claim held under reproduction
- 🟡Could not use the authors’ exact input data
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡The deviation was non-trivial in magnitude
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Software paper (RNAmountAlign RNA aligner). Described well enough to reproduce: the repo (github clotelab/RNAmountAlign @2a9579e = Zenodo 3595547) ships the authors' own C/C++ tool AND the benchmark input data (BRAliBase 2.1 K2, Rfam local set) as git-LFS objects; built cleanly with make (bundled libRNA.a) on «our HPC». The accuracy-SCORING script is NOT shipped, so we re-implemented the metric exactly as the paper's Methods define it (Sen=TP/refpairs, PPV=TP/predpairs, F1=harmonic mean; local coords recovered from CLUSTAL end-positions + FASTA-header reference coords). RESULT = 1:1 on the headline accuracy claims: pairwise GLOBAL F1 pooled 0.852 vs reported 0.84 (8587/8976 instances; per-family tRNA 0.778 vs 0.75, 5S 0.858 vs 0.85, 5.8S 0.920 vs 0.90, HIV_PBS 0.999 vs 1.00, Cobalamin 0.591 vs 0.56 — all within ~0.03, same family ranking); pairwise LOCAL PPV pooled 0.560 vs reported 0.53 (and 5S 0.385 vs 0.45). Small systematic +0.01..0.03 bias in global F1 consistent with the unshipped scoring script's exact convention. DISCREPANCY: local predicted alignment LENGTH 24.3 vs reported 50.35 (~2x shorter, flagged mismatch) — likely tool-version or the shipped local set being 2100 instances (100x21 fam) vs the paper's stated 1500 (75x20); PPV still matches so not a fabrication signal. 389/8976 global instances (4.3%) excluded because the tool rejects non-ACGU IUPAC ambiguity codes (clean tool limitation, documented). NOT ATTEMPTED: multiple-alignment SPS/SCI (BRAliBase K5 data not shipped), genome-scan runtime (hw-dependent), and all competitor-tool columns (out of scope - only the authors' own tool reproduced). No value appears fabricated; every reproduced number is derivable from the shipped tool+data and lands on the reported value.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 71assessed: 2026-06-16 ⛓ 66ad5e5fdbeb
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-16
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-18no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusBecause RNA secondary-structure distance metrics show poor inter-ensemble correlation and most existing alignment algorithms rely on base-pair/string-edit distance or free energy, can a fast sequence/structure alignment method based on (incremental ensemble) mountain distance achieve good accuracy with faster run time across local, global, semiglobal, and multiple alignment modes?
- ★ RNAmountAlign is the first RNA sequence/structure pairwise alignment algorithm based on incremental ensemble mountain distance, running in O(n^3) time and O(n^2) space for two sequences of length n. method
- ★ RNAmountAlign supports pairwise local, global, and global/semiglobal (query search) alignment plus progressive multiple alignment, using a weighted combination of sequence and structural similarity. method
- ★ RNAmountAlign returns p-values (and corresponding E-values) via Karlin-Altschul statistics for local alignment and via parameter fitting to normal, extreme value, and gamma distributions. method
- ★ RNAmountAlign achieves reasonably good accuracy with faster run time than LocARNA, LARA, FOLDALIGN, DYNALIGN, STRAL, MXSCARNA, and MUSCLE while supporting all alignment types. finding
- ★ The extension RNAmountAlignScan scans a target genome for hits with high sequence/structural similarity to a query, outperforming RSEARCH and sequence-only scans and running faster than FOLDALIGN query scan. method
- ★ RNA secondary structure distance metrics are highly correlated intra-ensemble but poorly correlated inter-ensemble, motivating a mountain-distance-based approach. finding
- ★ Incremental ensemble mountain height represents the Boltzmann structural ensemble of a length-n RNA as a length-n real vector computable in quadratic time given base pairing probabilities, enabling fast sequence-alignment methods. mechanism
- RNAmountAlign is provided as open-source C++ software with a web server. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Pairwise global sequence/structure alignment benchmarking (F1 accuracy) | RNA sequences from BRAliBase 2.1 K2 database | none | F1 score (harmonic mean of sensitivity and PPV) vs reference alignments | BRAliBase 2.1 K2 |
| Multiple global sequence/structure alignment benchmarking (SPS accuracy) | RNA sequence sets | none | SPS (sum-of-pairs score: correctly aligned nucleotide pairs / total reference aligned pairs) | — |
| Single pairwise alignment case study | 72 nt tRNA AL671879.2 and 69 nt tRNA D16387.1 (BRAliBase 2.1 K2) | none | agreement with reference alignment / ensemble mountain height display | RNAmountAlign web server (gap init -3, gap ext -1, structural weight gamma=1/2) |
| Whole-genome query scan (RNAmountAlignScan) | target genome sequence with single RNA query | none | hits with high sequence and structural similarity; comparison vs RSEARCH/FOLDALIGN/sequence-only scan | — |
| Base pairing probability / partition function computation | RNA sequence | none | base pairing probabilities p_ij and incremental ensemble mountain height | Vienna RNA package export_bppm() (energy parameters from ref [35]) |
| Consensus structure generation | alignment of tRNA AL671879.2 and D16387.1 | none | consensus secondary structure | RNAalifold (Vienna RNA Package) |
- – RNAmountAlign attains average pairwise global F1 score of 0.84, tied with LARA and MXSCARNA and highest among benchmarked tools F1=0.84
- – RNAmountAlign attains average multiple global SPS of 0.81, competitive with LocARNA (0.86), LARA (0.85), MXSCARNA (0.84) SPS=0.81
- – For the tRNA AL671879.2 / D16387.1 pair, the RNAmountAlign alignment is identical to the BRAliBase reference despite only 28% sequence identity (twilight zone) 28% sequence identity
- – Intra-ensemble correlation between metrics is high while inter-ensemble correlation is low (e.g., BP vs MD) intra r=0.822 vs inter r=0.210
- ▼ String edit (SE) vs tree edit (TE) correlation drops between intra- and inter-ensemble settings 0.975 to 0.590
- – Incremental ensemble mountain height m_a(k) is bounded in [-1,1] and computable in quadratic time given base pairing probabilities -1 <= m_a(k) <= 1
- correlation 0.822 (intra-ensemble Pearson correlation between base pair distance (BP) and mountain distance (MD))
- correlation 0.210 (inter-ensemble Pearson correlation between BP and MD)
- correlation 0.975 (intra-ensemble correlation between string edit (SE) and tree edit (TE) distance)
- correlation 0.590 (corresponding (inter-ensemble) correlation between SE and TE)
- other F1=0.84 (average pairwise global alignment F1 score for RNAmountAlign)
- other SPS=0.81 (average multiple global alignment SPS score for RNAmountAlign)
- other 28% (sequence identity of tRNA AL671879.2/D16387.1 pair correctly aligned by RNAmountAlign)
- other theta = 3 (steric constraint: minimum unpaired bases between paired positions (j - i > 3))
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a computational methods/software paper introducing RNAmountAlign, a C++ tool for RNA sequence/structure alignment. The primary evaluative approach is benchmark comparison: pairwise alignment accuracy is reported as average F1 score (harmonic mean of sensitivity and PPV) and multiple alignment accuracy as SPS (sum of pairs score) against several competing tools on the BRAliBase 2.1 K2 reference database. Statistical significance of alignment scores is provided through the software itself via Karlin-Altschul extreme-value statistics and parametric distribution fitting (normal, extreme value, gamma); no inferential hypothesis tests are applied to compare tool performance. Pearson correlation values characterizing inter- vs. intra-ensemble similarity of secondary structure metrics are reported from a previously published reference rather than computed in this paper.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Pearson correlation | Table 1 — comparison of inter-ensemble vs. intra-ensemble similarity of RNA secondary structure distance metrics (BP, MD, SE, TE, HTE) | — | not stated |
| F1 score (harmonic mean of sensitivity and PPV) | Table 2 — pairwise global alignment benchmarking against LocARNA, LARA, FOLDALIGN, DYNALIGN, STRAL, MXSCARNA on BRAliBase 2.1 K2 | — | na |
| SPS (sum of pairs score — fraction of correctly aligned nucleotide pairs vs. reference) | Table 2 — multiple alignment benchmarking | — | na |
| Karlin-Altschul extreme value distribution statistics | Local alignment E-value / p-value computation within RNAmountAlign software | — | not stated |
| Parametric distribution fitting (normal, extreme value, gamma) | RNAmountAlign parameter fitting for p-value estimation in local and global alignment | — | not stated |
-
Tool accuracy on the benchmark is reported as a single average F1 or SPS value with no measure of spread across the RNA families in the dataset↳ Could also: Bootstrap resampling over the benchmark RNA families could yield a 95% confidence interval (or standard deviation) around each tool's mean F1/SPS — Confidence intervals would convey how consistently each tool performs across diverse RNA families and allow informal assessment of whether observed differences in mean score are likely to be stable
-
Benchmarked tools are compared by tabulating point-estimate accuracy scores without a formal test of whether differences are statistically distinguishable↳ Could also: A Wilcoxon signed-rank test (or Friedman test with post-hoc correction) across per-family scores would formally compare tool performance while accounting for the paired structure of the benchmark — A paired non-parametric test acknowledges that all tools are evaluated on the same RNA families, and a multiplicity correction (e.g., Bonferroni or Benjamini-Hochberg) would control the false-discovery rate across the multiple pairwise tool comparisons
-
The Karlin-Altschul extreme value distribution (EVD) is applied for local alignment p-value computation, with the EVD parameters estimated analytically↳ Could also: Empirical null distributions could also be derived by randomly shuffling query sequences (preserving dinucleotide composition) and fitting the resulting score distribution — Dinucleotide-preserving permutation accounts for sequence composition biases not captured by the analytic EVD and is commonly used when the score distribution does not closely follow the theoretical extreme-value form
-
For the distribution fitting module, three parametric families (normal, EVD, gamma) are offered without a stated criterion for selecting among them for a given dataset↳ Could also: A likelihood ratio test or information criterion (AIC/BIC) comparing the three fitted distributions would also provide a principled selection rule — An explicit model-selection step helps users choose the best-fitting distribution for their particular alignment score distribution, reducing the risk of mis-specified p-values
-
Pearson correlation is used in the cited background work (Table 1) to summarize relationships among secondary structure distance metrics↳ Could also: Spearman rank correlation could also be used for the same comparison — Because the underlying distance distributions may be skewed or heavy-tailed, rank-based correlation is less sensitive to outliers and does not assume linearity of the relationship
-
The SPS metric counts aligned nucleotide pairs relative to the reference alignment without weighting by alignment length or family size↳ Could also: The Matthews Correlation Coefficient (MCC) or the modeler score could also be used as complementary benchmark metrics — MCC and related metrics account for both false positives and false negatives in a balanced way and are commonly used alongside F1/SPS in RNA alignment benchmarking to give a more complete picture of accuracy
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
- Rfam 15: RNA families database in 2025. L1 68/100
- Rfam 15: RNA families database in 2025. L1 68/100
- Rfam 15: RNA families database in 2025. L1 68/100
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-31978147 (RNAmountAlign)
Paper: Bayegan AH, Clote P. RNAmountAlign: Efficient software for local, global, semiglobal pairwise and multiple RNA sequence/structure alignment. PLoS ONE 2020. DOI 10.1371/journal.pone.0227177. PMCID PMC6980424.
Nature of paper: software/methods paper. The repo (github.com/clotelab/RNAmountAlign,
Zenodo 10.5281/zenodo.3595547) ships the C/C++ tool itself plus the benchmark input
data (BRAliBase 2.1 K2/K5, Rfam 12.0 local set, E. coli tRNA scan) as git-LFS objects.
Pinned commit: 2a9579ecac6cb612dfb12231eb99be502d736ee8 (2019-12-30, = Zenodo v1.0).
In scope (pipeline-derived; reproduced by running the authors' own tool)
The tool is deterministic; its accuracy/runtime numbers are the directly reproducible pipeline outputs. We focus on RNAmountAlign's own reported numbers (Tables 4, 5 and the multiple-alignment SPS/SCI), which we can reproduce 1:1 without installing the competitor tools.
Default benchmark parameters (Table 4 caption): RIBOSUM85-60, γ=0.5, gap-init −3, gap-ext −1 — these are the tool's compiled defaults, so plain invocation reproduces them.
- C1 — Pairwise global F1, pooled (Table 4). Reported RNAmountAlign = 0.84. Run global alignment on BRAliBase 2.1 K2 (8,976 pairwise alignments, 36 families); score F1 vs each instance's reference alignment.
- C2 — Pairwise global F1, per-family (Table 4). tRNA 0.75±0.20, 5S_rRNA 0.85±0.13, 5.8S_rRNA 0.90±0.09, HIV PBS 1.00±0.01, Cobalamin 0.56±0.16.
- C3 — Pairwise local (Table 5). RNAmountAlign pooled PPV=0.53±0.42, runtime
2.30±2.12 s, aln length 50.35±42.33; 5S_rRNA PPV=0.45±0.46, len 34.79±25.44.
Run
-localon the Rfam 12.0 local set (1,500 alignments, 75/family). - C4 — Multiple alignment SPS/SCI (results text). RNAmountAlign SPS=0.81±0.18, SCI=0.84±0.24 on BRAliBase 2.1 K5 (2,405 alignments of 5 sequences).
Scoring metric (paper §Methods)
- Sensitivity = TP / (reference alignment length); PPV = TP / (predicted alignment length).
- TP = number of correctly aligned residue pairs (aligned pairs present in both predicted and reference alignment). "Alignment length" = count of aligned (non-gap:non-gap) pairs.
- F1 = harmonic mean = 2·Sen·PPV/(Sen+PPV).
- The scoring script is NOT shipped in the repo — we implement this standard metric ourselves and document it. (Possible-divergence note in AUDIT.md.)
Out of scope / not attempted (or secondary)
- Competitor-tool numbers in Tables 2/4/5 (LocARNA, LARA, FOLDALIGN, DYNALIGN, STRAL, MXSCARNA, mLocARNA, Multilign, RSEARCH): require building many separate third-party tools; not the authors' code. Not attempted — we reproduce only RNAmountAlign's column.
- C5 — Genome query scan runtime (RNAmountAlignScan on E. coli, reported 1.63 h): attempted opportunistically if time allows; runtime is hardware-dependent so not a clean 1:1 numeric target.
- p-value distribution fitting (2.5M semiglobal scores, Fig figures): statistical-fit figures, secondary; not the primary accuracy claims.
Pipeline
Single tool: RNAmountAlign (C++/C, links bundled libRNA.a = ViennaRNA). Build with
make in src/. No external pipeline; benchmark data bundled in repo via git-LFS.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
The headline accuracy claims reproduce 1:1 within tolerance by building the authors' own shipped tool on the shipped data: global F1 0.852 vs 0.84 and local PPV 0.560 vs 0.53, with all per-family values within ~0.03 and the same ranking — nothing looks fabricated or too-perfect, so q5/q7 are green. The deviations are explainable and mostly on the authors'/data-availability side: the scoring script was not deposited (forcing a faithful re-implementation, likely causing the small +F1 bias), the local set is 2100 vs the stated 1500, and the local alignment length is ~2x short (24.26 vs 50.35) though PPV still matches. C4 (K5 SPS) and C5 (genome-scan runtime) could not be attempted (data not shipped / hw-dependent). Overall a solid reproduction with moderate, well-documented anomalies → yellow.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.