RNASEQR--a streamlined and accurate RNA-seq sequence analysis program.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- ✓Overall, the reproduction was clean
- 🟡A deviation arose in the data or preprocessing
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
RNASEQR (PMID 22199257) pipeline was successfully rebuilt from source and run to completion on the full paper-scale CCDS 75bp simulated benchmark (38.5M reads) and both real GSE33328 samples (SRR359290 peripheral, SRR359291 tumor; ~23.3M and ~25.4M reads). Three independent bugs were root-caused and fixed during setup: a self-introduced colorspace-flag patch bug (reverted to original code), lost executable permissions on bundled bowtie/blat binaries, and a chromosome-naming mismatch in the annotation-BED consumer requiring the original UCSC-style BED rather than the renamed one used for transcriptome-FASTA construction. After fixing these, an independent ground-truth-based accuracy analysis (parsing all 37.6M CCDS alignment records against read-name-encoded truth) reproduced the paper's headline claims closely: unique-mapping rate 97.73% (paper 97.75%), unspliced sensitivity 97.53% (paper 97.54%, effectively exact), spliced sensitivity 97.98% (paper 97.98%, exact), and overall position-accuracy 99.98% (paper 99.91%). Real-data unique-mapping rates (74.1%/76.4%) ran a few points below the paper's ~78.3%, graded within-tol given cross-generation software/hardware differences. Stage 3 (BLAT anchor-and-realign fallback) was confirmed to engage and complete without error on real data, resolving an earlier open question about BLAT stability against the rebuilt .2bit index. Splice-specificity claims (99.94%/99.90%) were not independently reproduced under a confirmed-identical methodology and are graded partial rather than fabricated.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-07-28
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-07-31no human curator yet
- Last updated
- 2026-07-31
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan a three-step 'align and remove' mapping strategy that first aligns RNA-seq reads to an annotated transcriptomic reference, then to the genome, then resolves novel junctions by anchor-and-align, produce more accurate RNA-seq read mapping than existing tools? The authors test whether RNASEQR built on this strategy outperforms ERANGE, MapSplice, SpliceMap and TopHat in mapping accuracy and downstream analyses.
- ★ RNASEQR is a new RNA-seq mapper/aligner that combines a BWT-based (Bowtie) transcriptomic/genomic alignment with hash-based BLAT local alignment in three sequential steps: transcriptome mapping, novel exon detection, and anchor-and-align novel splice junction identification. method
- ★ RNASEQR maps more reads uniquely and with higher overall accuracy than ERANGE, MapSplice, SpliceMap and TopHat on a simulated CCDS-derived 75-bp library. finding
- ★ RNASEQR is particularly effective for spliced reads spanning exon junctions, with both higher sensitivity and specificity than the other tools, and its advantage should increase with read length. finding
- ★ Accurate alignment by RNASEQR strongly reduces false single nucleotide variant calls, especially spurious SNVs near splice junctions that arise from incorrect junction identification in other tools. mechanism
- ★ Other tools underestimate gene-expression levels, predominantly for low-abundance protein-coding transcripts and for transcripts containing exons shorter than the read length. finding
- The choice of transcriptomic annotation database (Ensembl, UCSC, RefSeq, CCDS) does not significantly affect RNASEQR's overall mapping performance. finding
- RNASEQR is an open-source pipeline (Python 2.7, 64-bit Linux) whose SAM-format output supports a wide variety of downstream analyses on desktop computers, available at https://github.com/rnaseqr/RNASEQR. resource
- RNA-seq data from a glioblastoma tumor and matched peripheral brain tissue were generated and deposited in GEO under accession GSE33328. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| In silico simulated RNA-seq mapping benchmark (75-bp reads generated by sliding a 1-nt-step window over full-length CCDS transcripts) | Human CCDS Gene transcripts assembled on human genome reference GRCh37 | none | Uniquely mapped reads, sensitivity and specificity for unspliced and spliced reads, overall mapping accuracy, incorrectly mapped reads | RNASEQR vs ERANGE v3.2.1, MapSplice v1.14.1, SpliceMap v3.3.5.1, TopHat v1.1.1 |
| Bulk poly(A) RNA-seq, 75-bp single-end | Glioblastoma multiforme (GBM) tumor tissue and matched peripheral brain tissue from one patient (Chang Gung Memorial Hospital, Taiwan) | other (tumor vs matched peripheral normal brain tissue) | High-quality sequence reads; uniquely/multiply mapped and unmapped read fractions; spliced vs unspliced reads | Illumina Genome Analyzer II with SBS sequencing kit v2; miRNeasy mini kit (Qiagen) for RNA extraction |
| RNA quality control / bioanalysis of total RNA and cDNA libraries | Total RNA from 0.5 g frozen GBM tumor and peripheral brain tissue | none | RNA integrity and cDNA library size distribution (~200-bp size-selected fragments) | Agilent 2100 Bioanalyzer |
| Mapping with different transcriptomic reference annotations | Two experimental human GBM RNA-seq libraries; transcriptomic indexes from Ensembl, UCSC KnownGene, NCBI RefSeq, CCDS on GRCh37 | other (annotation database swapped) | Reads uniquely/multiply mapped per pipeline step and total uniquely mapped percentage | Bowtie-built transcriptomic indexes; genomic index from Bowtie webpage |
| SNV calling from alignment output | Simulated CCDS-derived library (no sequence variation introduced) and experimental GBM libraries | none | Number and genomic location of false SNVs (coding exon, non-coding exon, intron ≤5 bp or >5 bp from exon, intergenic); SNVs with read depth <5 removed | SAMtools v0.1.8 |
| Transcript assembly, isoform determination and gene-expression quantification | Mapped reads from GBM tumor and peripheral brain libraries; Ensembl-annotated transcripts | none | RPKM expression levels, Pearson correlation of expression between tools, transcripts with >2-fold expression differences, novel exons and novel splice junctions | Scripture |
| Exon-level detection analysis | Human gene transcripts with expression >1 RPKM in GBM tumor and peripheral brain, e.g. ST13 (12 exons, 5 of them <76 bp) | none | Number/ratio of unidentified exons stratified by exon length (<76 bp vs ≥76 bp) | Integrative Genomics Viewer for visualization |
- ▲ On the simulated CCDS library, RNASEQR uniquely mapped 37 634 842 of 38 506 959 reads (97.74%), more than ERANGE (96.42%), MapSplice (97.28%), SpliceMap (89.64%) and TopHat (92.89%), with the highest overall mapping accuracy of 99.91% 97.74% uniquely mapped; 99.91% overall accuracy vs 96.95/96.69/87.48/90.50%
- ▲ For spliced reads, RNASEQR achieved 97.98% sensitivity and 99.90% specificity, far above ERANGE (88.41%), MapSplice (90.26%), SpliceMap (69.29%) and TopHat (65.36%) sensitivity sensitivity 97.98% vs 65.36–90.26%
- ▼ RNASEQR produced the fewest incorrectly mapped reads (32 184) versus 1 133 128–4 322 547 for the other tools, and 86.31% of its incorrect reads were partially correct 32 184 vs up to 4 322 547 (~35–134-fold fewer)
- ▼ RNASEQR yielded the fewest false SNVs, e.g. 719 in coding exons and 330 in introns ≤5 bp from an exon, versus 27 090–126 812 and 249 573–704 682 respectively for the other tools 330 vs 249 573–704 682 junction-proximal false SNVs
- – On experimental GBM data RNASEQR assigned ~78.3% of reads to unique genomic regions, 2.1% to multiple locations, and left 20.8% unmapped (mostly low Phred quality reads); ~3% more sequences than other programs, though TopHat mapped 0.5% more by tolerating more mismatches and truncating low-quality reads 78.3% unique; ~3% more than others; TopHat +0.5%
- ▲ RNASEQR mapped 7.3 million spliced sequences from the experimental libraries, 17–106% more than the other tools 17–106% more spliced reads
- – Different transcriptomic annotation databases changed the number of uniquely mapped sequences by only 1.71% (75.38–77.09% total uniquely mapped); with CCDS only about one-third of reads were assigned in step I but equivalent totals were reached after steps II and III 1.71% difference
- – RNASEQR was the only tool able to detect all 12 exons of ST13 (5 exons <76 bp), and showed a significantly lower ratio of unidentified exons for exons <76 bp; it still failed to detect 4658 exons (peripheral brain) and 4762 exons (tumor) among transcripts >1 RPKM 4658 and 4762 undetected exons
- count 38 506 959 non-redundant simulated 75-bp sequences (23 187 354 unspliced, 15 319 605 spliced) (Simulated CCDS-derived benchmark library composition)
- other Overall mapping accuracy 99.91% (RNASEQR), 96.95% (ERANGE), 96.69% (MapSplice), 87.48% (SpliceMap), 90.50% (TopHat) (Tool comparison on simulated library)
- other Spliced-read sensitivity/specificity 97.98%/99.90% (RNASEQR) vs 88.41%/99.85% (ERANGE), 90.26%/99.50% (MapSplice), 69.29%/88.90% (SpliceMap), 65.36%/99.15% (TopHat) (Splice-junction mapping performance)
- count Incorrectly mapped reads: 32 184 (RNASEQR), 1 133 128 (ERANGE), 1 240 521 (MapSplice), 4 322 547 (SpliceMap), 3 399 156 (TopHat) (Mis-mapping on variation-free simulated data)
- count 25 384 704 tumor and 23 258 943 reference single-end 75-bp reads (25.3 and 23.2 million high-quality sequences; 48 643 647 total) (Experimental GBM patient RNA-seq libraries)
- count Total mapped uniquely: 35 276 746 (76.41%) Ensembl, 35 588 871 (77.09%) UCSC, 35 483 393 (76.86%) RefSeq, 34 800 975 (75.38%) CCDS; differing by only 1.71% (Effect of transcriptomic annotation on RNASEQR output)
- fold_change 21 549 Ensembl transcripts showed ≥2-fold expression change in glioblastoma tumor versus peripheral brain tissue (Differential expression from RNASEQR + Scripture (Supplementary Table S1))
- count Annotated transcripts per reference: 151 185 (Ensembl), 77 614 (UCSC), 37 162 (RefSeq), 23 754 (CCDS) (Sizes of the four transcriptomic reference databases)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This paper describes a computational tool benchmarking study rather than a hypothesis-testing biological experiment. RNASEQR's mapping performance was compared to four other RNA-seq alignment tools (ERANGE, MapSplice, SpliceMap, TopHat) using a simulated CCDS-derived read dataset with known true positions, and using two experimental RNA-seq libraries (tumor and matched peripheral brain tissue from one glioblastoma patient). Results were reported as descriptive performance metrics (sensitivity, specificity, overall mapping accuracy, counts of uniquely/multiply mapped reads, SNV counts) and as Pearson correlation coefficients and fold-change comparisons of gene-expression estimates between tools, without inferential hypothesis tests.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Pearson's correlation coefficient | Comparison of gene-expression level (RPKM) estimates between RNASEQR and other mapping tools, Figure 3A and B | Ensembl-annotated transcripts with expression levels >1 RPKM (exact n not stated) | not stated |
| Sensitivity/specificity/overall accuracy calculation (descriptive, not an inferential test) | Simulated CCDS-derived 75-bp read dataset, Table 2, comparing RNASEQR to ERANGE, MapSplice, SpliceMap and TopHat | 38 506 959 simulated sequences (23 187 354 unspliced, 15 319 605 spliced) | na |
| Fold-change comparison (2-fold threshold, descriptive) | Gene-expression level comparison between glioblastoma tumor and peripheral brain tissue, and between RNASEQR and other tools, Figure 3C and D, Supplementary Table S1 | 21 549 Ensembl transcripts stated as showing 2-fold change (tumor vs. peripheral brain) | na |
-
Tool performance is summarized with point-estimate percentages (sensitivity, specificity, overall mapping accuracy) without any measure of uncertainty.↳ Could also: Bootstrap resampling of reads or read-depth-stratified subsampling to generate confidence intervals around these accuracy metrics — This would convey how much the reported percentages might vary if a different random subset of reads or a different simulated library were used, complementing the single point estimates presented.
-
Differential expression between tumor and peripheral brain tissue is assessed using a fixed 2-fold change threshold on RPKM-based estimates without an accompanying statistical significance test.↳ Could also: Count-based differential expression frameworks such as DESeq2 or edgeR, which model read-count variance and report a Wald or likelihood-ratio test p-value alongside fold change, combined with FDR correction — This would allow fold-change magnitude to be considered jointly with statistical confidence, which can be useful when comparing single-sample tumor/normal pairs where technical variance is otherwise unaccounted for.
-
The comparison between RNASEQR and other tools is based on a single simulated dataset and a single GBM patient's paired tumor/normal samples.↳ Could also: Including additional patient samples or replicate libraries — This would allow assessment of how consistently the observed mapping performance differences generalize across different patients or tissue preparations, beyond the one dataset and one patient presented here.
-
Gene-expression concordance between tools is summarized using Pearson's correlation coefficient (Figure 3A, B), which is sensitive to the low-abundance outlier transcripts the paper itself highlights.↳ Could also: Spearman's rank correlation coefficient, computed alongside or instead of Pearson's — Rank-based correlation is less influenced by extreme or outlier values, such as the low-abundant transcripts noted as underestimated by other tools, and can offer a complementary view of monotonic association.
-
SNVs with read depth below five were removed using a fixed manual threshold rather than a probabilistic variant-calling model.↳ Could also: A likelihood- or Bayesian-based variant caller (e.g., GATK HaplotypeCaller or samtools/bcftools genotype likelihoods) that incorporates base quality and depth into a formal confidence score — This would let the variant filtering incorporate a statistical confidence measure rather than a single hard depth cutoff, which some readers may find informative when interpreting the SNV counts in Table 3.
-
No correction for multiple comparisons is described despite many parallel per-transcript or per-tool comparisons being reported.↳ Could also: A false discovery rate procedure such as Benjamini-Hochberg applied across the set of per-transcript comparisons — This is a standard way to control the expected proportion of false positives when many individual comparisons (e.g., per-gene fold-change calls) are examined simultaneously.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
Near-1:1 reproduction of a tool paper. RNASEQR was rebuilt from the authors' own source and rerun at full paper scale (38,506,959 reads of the bundled CCDS 75bp benchmark plus both real GSE33328 samples), and an independent ground-truth parse of all 37.6M alignment records reproduced the headline claims to within 0.02 pp: unique mapping 97.73% vs 97.75%, unspliced sensitivity 97.53% vs 97.54%, spliced sensitivity 97.98% vs 97.98% exact. The residual deviations are all on the input/environment side and firmly attributable to 14 years of toolchain drift — real-data unique mapping 74.12%/76.45% vs ~78.3% after a from-scratch reference build, and a 6,403-read (0.017%) difference in the distributed benchmark FASTQ. The only genuinely unresolved item is the specificity pair (99.94%/99.90%), which the paper does not define precisely enough to rebuild a true-negative denominator; the reproducers correctly flagged it as partial rather than manufacturing a match, and the three bugs they root-caused (missing +x bit on bundled binaries, chr-naming in the -f annotation BED, a self-introduced patch) were packaging/setup issues, not defects in the published result. Overall: green — the reported values are derivable, the core performance claim holds at scale, and the one gap is an underspecified-method note on the authors' side rather than a substantive discrepancy.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.