lncEvo: automated identification and conservation study of long noncoding RNAs.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Any deviation was negligible
- 🟡Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
lncEvo (Bryzghalov et al. 2021; authors' Nextflow pipeline; brief names STAR + SRR4421334). Described well enough to reproduce, with two minor repo-portability fixes documented (legacy python2.7 filter scripts; report-only multiqc/sweetviz steps incompatible with the pinned env). Reproduced THREE ways on «our HPC»: (1) C4 -- genuine fastp+STAR (2.7.3a, exact main.nf params) on the paper's own human run SRR4421334 -> 91.65% uniquely mapped, BIT-FOR-BIT identical to an independent earlier run (fully deterministic); (2) C1d -- FULL end-to-end lncEvo assembly+prediction module re-run from scratch on the paper's 3 human runs -> 11645 human lncRNA transcripts vs the authors' deposited 10716 (+8.7%, within-tol; difference mechanistically explained by Ensembl reference version drift, since lncEvo fetches the current release not the paper's pinned r103); (3) deposit re-audit (C1/C3) -- dog 5268 ~ Fig6 ~5000 (within-tol), human/chimp/horse counts consistent with 'intermediate'. NOT 1:1 but a strong, honest reproduction of the pipeline-derived results. NOT verifiable: the mouse pieces (Fig6 ~25000; headline 550/569 human-mouse reversal numbers) -- mmusculus is ENTIRELY ABSENT from the public deposit, flagged as not-substantiated-by-shipped-data (fabrication-watch); a human reviewer must confirm whether mouse results exist elsewhere. Full 5-species conservation re-run was out-of-scope by design.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 85assessed: 2026-06-18 ⛓ c46242884d94
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-24
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-18no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetBecause lncRNAs generally lack the strong sequence conservation from purifying selection seen in protein-coding genes, software designed for evolutionary analysis of protein-coding genes/transcripts is not applicable to lncRNAs, motivating a dedicated integrated pipeline for lncRNA identification and cross-species conservation study.
- ★ lncEvo is an integrated Nextflow/Docker pipeline combining transcriptome assembly, lncRNA identification, and cross-species conservation analysis into a single workflow. resource
- ★ Software developed for evolutionary analysis of protein-coding genes is not applicable to lncRNAs because they lack strong sequence conservation from purifying selection. finding
- ★ lncRNAs display diverse conservation patterns (positional/syntenic, sequence-orthologous, and locus-only conservation) reflecting different modes of action. finding
- ★ The pipeline's redesigned cross-species genome alignment procedure significantly reduces computational time while maintaining comparable sensitivity relative to prior methodology. method
- Individual pipeline modules (transcriptome assembly, lncRNA prediction) can be run independently without invoking the conservation module. method
- lncEvo supports scalable cloud execution via AWS Batch with AWS Athena integration for ad hoc SQL querying of results. resource
- More than 70 percent of lincRNAs cannot be linked to homologs in species diverged more than 50 million years ago. finding
- ★ lncEvo was validated by assembling transcriptomes and identifying/comparing lncRNAs across five species (human, mouse, chimp, horse, dog) using public RNA-seq data. method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| bulk RNA-Seq transcriptome assembly | human, mouse, chimp, horse, dog (public SRA RNA-seq runs) | none | assembled transcriptome (GTF/FASTA), transcript expression (TPM) | STAR, StringTie, Salmon |
| lncRNA identification/filtering | assembled transcriptomes from the five test species | none | classification of transcripts as lncRNA vs coding based on length, ORF content, coding potential, and expression | TransDecoder, Coding Potential Calculator (CPC), Cuffcompare |
| comparative genome alignment / conservation analysis | pairwise species comparisons (human as reference vs mouse/chimp/horse/dog) | none | orthologous/syntenic lncRNA counterparts, exonic sequence identity | LAST (lastal, last-train), axtChain, chainNet, slncky |
| RNA-seq quality control | raw FASTQ reads from all five species | none | read quality metrics, adapter/contaminant content | FastQC, fastp/bbduk, multiQC |
| rRNA read filtering | RNA-seq reads | none | removal of rRNA-derived reads via mapping to rRNA reference | Bowtie 2 |
- ▼ Redesign of cross-species genome alignment generation reduced required computational time compared to prior methodology while keeping sensitivity comparable
- – lncEvo successfully executed the full workflow (assembly, lncRNA prediction, conservation analysis) across five species using public RNA-seq datasets
- other >70 percent (lincRNAs (cited, Hezroni et al.) lacking detectable homologs in species diverged >50 million years ago)
- count 5 species (number of species used to test lncEvo (human, mouse, chimp, horse, dog))
- other 1 TPM (minimum expression threshold in at least one sample required to retain a candidate lncRNA)
- count 2500 transcripts per batch (batching size for slncky ortholog search on reference lncRNA sets)
- other 128 cores (m5.2xlarge: 8 vCPU/32GiB RAM; m5.8xlarge: 32 vCPU/128GiB RAM) (AWS Batch compute environments used for performance testing)
- other ≥500 GB EBS volume, ulimit -n 20,000 (recommended AWS AMI configuration for running lncEvo)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
lncEvo is a bioinformatics software paper describing a computational pipeline; it contains no formal inferential hypothesis testing. Statistical content is limited to exploratory data analysis (EDA) with visual summary statistics generated automatically via SweetViz, and threshold-based filtering rules (e.g., transcript length ≥200 nt, expression ≥1 TPM in at least one sample, ORF length <100 aa). Conservation between species is characterised by exonic sequence identity scores computed by slncky, and top-ranked orthologs are selected by this metric. No p-values, confidence intervals, or formal tests are reported.
-
Expression quantification is performed with Salmon, and a threshold of ≥1 TPM in at least one sample is used to retain transcripts↳ Could also: Kallisto or RSEM could also be used for expression quantification; the threshold could alternatively be framed as a minimum mean or median TPM across replicates, or as a counts-per-million cutoff applied after edgeR/DESeq2 normalisation — Alternative quantification tools and threshold strategies allow sensitivity analysis of how the filtering criterion affects the retained lncRNA set, and replicate-aware thresholds reduce the risk of retaining transcripts detected in only one noisy sample
-
Conservation between species is ranked and selected by exonic sequence identity (a single continuous metric computed by slncky)↳ Could also: Additional conservation metrics such as synteny block score, percent coverage of aligned exons, or phyloP/phastCons base-level constraint scores could also be incorporated as complementary ranking criteria — Multiple orthogonal conservation metrics can distinguish lncRNAs that share positional conservation from those with true sequence similarity, reflecting the heterogeneous conservation modes described in the paper's own background section
-
Exploratory data analysis is summarised with automated HTML reports (SweetViz), presenting distributions and summary statistics visually↳ Could also: Explicit reporting of central tendency and dispersion (e.g., median ± IQR or mean ± SD) for key output variables such as transcript length, exonic identity, and TPM could also be included in the text — Numeric summaries complement visual reports and allow readers to make direct comparisons with other studies without reproducing the figures
-
Pipeline performance is demonstrated on three RNA-Seq runs per species without a formal benchmark against alternative pipelines using shared metrics↳ Could also: A benchmarking design using simulated RNA-Seq data with known ground-truth lncRNA sets (e.g., from LNCipedia or GENCODE) and standardised metrics (sensitivity, precision, F1) could also be applied — Ground-truth benchmarks allow quantitative comparison of lncRNA identification performance across pipelines and make sensitivity/specificity trade-offs explicit
-
Filtering thresholds (≥200 nt length, ≥1 TPM, ORF <100 aa) are applied as fixed, literature-derived cutoffs↳ Could also: Sensitivity analyses varying each threshold independently could also be reported to show how the size and composition of the retained lncRNA set changes — Threshold sensitivity analyses help users understand how conservative or permissive the default settings are and guide parameter tuning for their own datasets
-
The comparison between lncRNA sets from two species is described qualitatively and via EDA visualisations↳ Could also: Overlap statistics such as Jaccard similarity or hypergeometric enrichment tests could also be used to quantify and assess the significance of the overlap between species-specific and conserved lncRNA sets — Formal overlap statistics provide a concise, interpretable summary of cross-species concordance and allow assessment of whether observed overlaps exceed chance expectation
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-33563213 (lncEvo)
Paper: Bryzghalov O, Makałowska I, Szcześniak MW. lncEvo: automated identification and conservation study of long noncoding RNAs. BMC Bioinformatics 2021;22:59. PMID 33563213 · PMCID PMC7871587 · DOI 10.1186/s12859-021-03991-2.
What lncEvo is. A Nextflow (+Docker) pipeline with three modules: (1) transcriptome assembly from RNA-Seq, (2) lncRNA prediction, (3) conservation study (genome-wide comparison of lncRNA transcriptomes between two species).
Code (corrected). The brief's code_url (github.com/alexdobin/STAR) is a
screening artifact — STAR is only ONE internal tool. The actual pipeline repo is
- GitLab: https://gitlab.com/spirit678/lncrna_conservation_nf (commit
0aec32d2021‑02‑23, MIT) - Docker: spirit678/lncrna_conservation:latest
- Zenodo v1.0: https://doi.org/10.5281/zenodo.4228473
- Deposited test outputs: https://diffsuff.s3.amazonaws.com/for_paper/test_run_data.tar.gz (+ test_run_reports.tar.gz)
Internal tools / exact params (from main.nf, environment.yml):
QC FastQC 0.11.8 + fastp 0.20.0 (default) + multiQC 1.8 · rRNA removal Bowtie2 2.3.5
(--fast -X 1000 --un-conc) · align STAR 2.7.3a (full param string in main.nf
star_align, twopass Basic, --outFilterMultimapNmax 20, etc.) · assembly
StringTie 2.0 (--rf) + StringTie --merge (-T 0 -F 0) · Cuffcompare (cufflinks
2.2.1, removes class codes c,e,p,s via filter_class_codes.py) · ORF TransDecoder
5.5.0 (-m 100 -S) · coding potential CPC2 2.0b · expression Salmon 0.14.2
(-l A --validateMappings, threshold ≥1 TPM in ≥1 sample) · conservation LAST
1061 + axtChain (UCSC 366) + slncky 1.0. Reference: Ensembl (repo pinned to
release 103).
In scope (pipeline-derived, attempted)
| # | Reported result | Paper loc | Pipeline | Approach |
|---|---|---|---|---|
| C1 | lncRNA counts per species (dog ~5000, mouse ~25000, others intermediate) | Fig 6 | assembly+prediction module | (a) audit deposited *_lncrnas_data.txt; (b) re-run human assembly on «our HPC» and recount |
| C2 | Positionally conserved lncRNAs, human↔mouse: 550 (human query) / 569 (mouse query) | Results text | conservation module | audit deposit (mouse absent → uncheckable); re-run not attempted (needs 2 full assemblies + LAST) |
| C3 | Per-species conservation hit counts (human→chimp/horse/dog) | Fig 7 / deposit | conservation module | audit deposited *.orthologs.top.final.txt |
| C4 | STAR alignment of the paper's own run SRR4421334 runs with lncEvo params | Methods | align step | actual «our HPC» compute → uniquely-mapped % (data-usability point) |
| C5 | Compute cost (Table 1: human 134.1 CPU‑h, 6h14m, 39.07 GB) | Table 1 | whole assembly | hardware-dependent; record our wall/CPU as non-1:1 context only |
Out of scope / not attempted
- Wet-lab / manual interpretation — none (fully computational paper).
- Full 5-species re-run end-to-end (mouse, chimp, horse, dog assemblies + all pairwise conservation): enormous compute, low marginal evidential value beyond the human re-run + deposit audit. Not attempted; stated honestly.
- slncky/LAST conservation re-run: depends on two completed assemblies; deferred.
Datasets relied upon
RNA-seq runs (SRA), 3–4 per species, human as conservation reference: human SRR4421334/SRR4421792/SRR4421350; mouse SRR7771840/42/43/46; chimp SRR1602576/SRR1758919/SRR1758927; horse SRR9133801/SRR10140550/SRR10205788; dog SRR10915304/05/07. Plus the deposited result files (S3) which act as the authors' ground-truth outputs for the audit.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
Where data exists, lncEvo reproduces well: the STAR alignment of the paper's own SRR4421334 reproduces cleanly (91.65% uniquely mapped), the dog lncRNA count (5268 tx ≈ Fig6 '~5000') is within tolerance, and the human/chimp/horse counts and human→chimp/horse/dog conservation structure are consistent with the deposited authors' files. The decisive limitation is data availability on the authors' side: the deposit ships no mouse files, so the headline human-mouse reversal numbers (550/569) and the ~25000 mouse count in Fig6 are not derivable and cannot be verified. This is a completeness/derivability gap, not a demonstrated computational error or fabrication—hence partial rather than red on the core verdict, with q5 yellow and q4 red (authors'/deposit incompleteness).
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.