Bayesian transcriptome assembly.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- 🔴Reported values were only indirectly comparable
- 🔴A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🔴The deviation was non-trivial in magnitude
- 🔴The central claim did not (fully) hold under reproduction
- 🔴Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
TopHat2/Bowtie2 alignment of the paper's H1 rep1 real RNA-seq data (SRR307911) succeeded and reproduced the described preprocessing methodology (88.5% overall mapping rate, 75.3% concordant pair rate). Bayesembler itself -- the paper's actual subject software -- could not be run to completion on this real data in either the paper's exact 'Publication release' (v1.1.1) or the only newer release (v1.2.0): a reproducible internal assertion-failure crash occurs in its core Gibbs-sampling stage after extensive infrastructure fixes (isolated compat-library LD_LIBRARY_PATH fix) and after refuting a specific root-cause hypothesis (CIGAR 'D'-operation scan across all 98.4M reads found zero matches). This crash is corroborated by multiple open, unanswered GitHub issues describing the same failure on independent datasets since 2015, indicating a genuine unfixed upstream defect rather than a local error. A substitute Cufflinks+gffcompare pathway was run in place of Bayesembler's own output (Flux Simulator, the paper's own benchmarking tool, was unavailable -- no Java runtime), producing plausible but non-equivalent transcript-assembly-concordance numbers against UCSC knownGene. All 5 real-data samples the paper's Table 1 relies on (K562 rep1/2, H1 rep1/2, mouse dendritic cells) were independently verified as accessible via ENA with observed read counts closely matching the paper's reported approximate depths; only H1 rep1 was carried through the full pipeline in this pass (80/20 scope decision). The paper's separate PacBio validation dataset (GSE51861) was confirmed to exist and match its description, but its exact run-level accession/read-count could not be resolved via the ENA API in the time available, and it is out of scope for the reproduced pipeline regardless (it feeds a different, non-Bayesembler sub-analysis in the paper).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-08-03
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-08-03no human curator yet
- Last updated
- 2026-08-03
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan transcriptome assembly from RNA-seq data be improved by casting it as Bayesian inference over a generative model of the RNA sequencing process, in which sparsity in the number of expressed transcripts is modelled explicitly without penalising lowly abundant transcripts? The authors test whether posterior sampling frequency of transcript candidates serves as a confidence metric that yields better sensitivity and precision than existing assemblers.
- ★ Bayesembler, a probabilistic transcriptome assembler built on a Bayesian model of the RNA sequencing process with Gibbs sampling over expressed candidates, abundances and read assignments, is introduced. method
- ★ Modelling candidate expression with binary Bernoulli-prior variables and a symmetric Dirichlet prior on abundances decouples expression from abundance, so sparsity is imposed without implicitly penalising lowly abundant transcripts (unlike regularisation-based assemblers). mechanism
- ★ The frequency at which a candidate is sampled as expressed during Gibbs sampling is a usable per-transcript confidence metric, enabling assembly selection and prioritisation of transcripts for downstream validation. mechanism
- ★ Bayesembler achieves higher sensitivity and precision than state-of-the-art assemblers (Cufflinks, IsoLasso, CEM, Traph) on both simulated and real RNA-seq data. finding
- ★ Improved performance is robust across species, tissues, library construction protocols and sequencing depths, and holds independent of transcript abundance level. finding
- ★ Bayesembler estimates the number of expressed splice variants per gene more accurately than other assemblers, which may partly explain its accuracy gains; Cufflinks markedly underestimates variant number per gene. finding
- Transcript length distributions produced by Bayesembler more closely resemble those of simulated/annotated transcripts, whereas other assemblers tend to produce shorter transcripts. finding
- Bayesembler is released as a complete C++ transcriptome assembly package, freely available at https://github.com/bioinformatics-centre/bayesembler. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Simulated paired-end strand-specific RNA-seq (~80 million reads) generated from the UCSC Known Genes annotation | in silico human transcriptome (40,496 annotated transcripts) | none | sensitivity (fraction of simulated transcripts assembled correctly), precision (fraction of assembled transcripts among simulated), counts of correct/incorrect transcripts, transcript length distribution, splice-variant number per gene, abundance accuracy | Flux Simulator |
| Paired-end strand-specific RNA-seq assembly benchmarked against the UCSC Known Genes annotation | human K562 erythroleukaemia cell line, two biological replicates | none | number of annotation-confirmed transcript predictions plotted against total predictions across abundance thresholds (curve height = sensitivity, slope = precision) | — |
| Paired-end strand-specific RNA-seq assembly benchmarked against the UCSC Known Genes annotation | human H1 embryonic stem cell line, two biological replicates (held-out test set) | none | number of annotation-confirmed predictions vs total predictions across abundance thresholds; transcript length distributions | — |
| Paired-end strand-specific RNA-seq assembly benchmarked against annotation | mouse dendritic cells, single replicate | none | number of annotation-confirmed predictions vs total predictions across abundance thresholds | — |
| Long-read validation of assembled transcripts | human H1 embryonic stem cells (both replicate assemblies), compared to previously published long-read data (Au et al.) | none | number of predicted transcripts verified by a PacBio read vs total predictions across abundance thresholds | Pacific Biosciences (PacBio) platform (Pacific Biosciences, California, USA) |
| Replicate-stability analysis of assemblies | human K562 and H1 cell lines, pairs of biological replicates | none | number of stable transcripts (multi-exonic transcripts identical between replicates) vs number of predicted transcripts across transcript-rank thresholds; replicate abundance correlations | — |
| Read mapping and splice graph construction (assembly pipeline input) | all RNA-seq datasets (human K562, human H1, mouse dendritic cells, simulated) | none | spliced read alignments; splice graphs; enumerated transcript candidates (capped at 100 per graph) | TopHat2 for mapping; graph construction routine from the CEM assembly package |
| Computational resource benchmarking | deepest benchmark RNA-seq dataset (~125 million paired-end reads) | none | wall-clock runtime and maximum memory footprint | 16 CPU cores (benchmark server had 40 CPUs) |
- ▲ On simulated data, Bayesembler assembled 3,528 more correct transcripts and produced 9,427 fewer incorrect transcripts than the runner-up assembler IsoLasso +3,528 correct; -9,427 incorrect
- ▲ Bayesembler showed higher sensitivity and precision than all other assemblers on simulated data, and this held across transcript abundance levels
- ▲ On real K562, H1 (both replicates) and mouse dendritic cell data, the Bayesembler annotation-confirmation curve extended higher and ascended more steeply than all other assemblers, indicating better sensitivity and precision
- ▲ For H1 replicate assemblies, Bayesembler verified more transcripts by PacBio long reads, both in absolute number and relative to total predictions at each abundance threshold
- ▲ Transcripts predicted by Bayesembler were more stable across biological replicates (K562 and H1) than those of any other assembler
- – Bayesembler had lower mean absolute difference between predicted and simulated number of splice variants per gene than other assemblers; Cufflinks markedly underestimated transcript number for many genes
- – Abundance estimates from Bayesembler showed marginally better Spearman rank agreement with simulated values than all other assemblers; on replicates, Bayesembler was best for K562 and Cufflinks best for H1, with only minor differences overall
- ▼ Assembly of the deepest dataset took ~5.5 hours on 16 CPU cores with a maximum memory footprint of 1.7 GB, faster and more memory efficient than Cufflinks 5.5 hours; 1.7 GB peak memory
- count 3,528 more correct transcripts assembled than IsoLasso (Bayesembler vs runner-up IsoLasso on simulated data)
- count 9,427 fewer incorrect transcripts than IsoLasso (Bayesembler vs runner-up IsoLasso on simulated data)
- count 40,496 annotated transcripts (number of UCSC Known Genes transcripts used as ground truth in the simulation)
- count approximately 80 million paired-end strand-specific reads (size of the simulated RNA-seq dataset (Flux Simulator))
- count approximately 125 million paired-end reads (deepest benchmark dataset used for runtime measurement)
- other ~5.5 hours on 16 CPU cores; 1.7 GB maximum memory (Bayesembler runtime and memory on the deepest benchmark dataset)
- other confidence > 0.5 and at least 12 expected paired-end read counts (thresholds for including a candidate transcript in the final assembly)
- count 100 transcript candidates maximum per splice graph (candidate pruning limit during graph traversal)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper describes a Bayesian probabilistic method (Gibbs sampling) for transcriptome assembly and evaluates it primarily through descriptive performance metrics rather than classical inferential statistics. Performance is reported as sensitivity and precision curves across abundance thresholds, comparing five assemblers on one simulated RNA-seq dataset and several real datasets (human K562, H1, and mouse dendritic cells, some with two biological replicates). Abundance estimation accuracy across assemblers is summarized with a single correlation coefficient (Spearman's rank correlation), and no p-values, confidence intervals, or formal hypothesis tests are reported in the provided text.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Spearman's rank correlation | Figure 2g — correlation between predicted and simulated transcript abundances | transcripts predicted correctly by all five assemblers (exact count not stated) | not stated |
| Mean absolute difference (descriptive summary, not an inferential test) | Figure 2f — difference between predicted and simulated number of splice variants per gene | — | na |
| Curve-based sensitivity/precision comparison across abundance or rank thresholds (descriptive metric, not an inferential test) | Figures 2a–d, 3a–c, 4a–c — comparisons of assemblers on simulated and real data using annotation-, PacBio-, and replicate-stability-based validation | — | na |
-
Abundance estimation accuracy across assemblers was summarized using Spearman's rank correlation between predicted and simulated abundances.↳ Could also: Pearson correlation or a Bland-Altman-style agreement analysis could also be used — These would additionally quantify the magnitude and direction of any systematic bias between predicted and simulated abundance values, complementing what a rank correlation shows about the monotonic relationship alone.
-
Comparisons among the five assemblers relied on descriptive sensitivity/precision curves without accompanying variability estimates or formal significance tests.↳ Could also: Bootstrap resampling over reads/transcripts, or a paired non-parametric test such as the Wilcoxon signed-rank test, could also be applied — This would provide confidence intervals or p-values around the performance differences, giving a formal statistical basis for comparing assemblers beyond the point estimates shown in the curves.
-
Real-data validation used two biological replicates for K562 and H1, and a single replicate for mouse dendritic cells.↳ Could also: Additional biological replicates, or a variance-component analysis leveraging replicate data, could also be used — More replicates per condition would allow explicit quantification of biological variability and how much it contributes to observed differences in assembler performance.
-
The difference between predicted and simulated number of splice variants per gene was summarized with a single mean absolute difference (Figure 2f).↳ Could also: Reporting the full distribution (e.g., median with interquartile range) or a paired non-parametric test could also be used — This would convey the spread of per-gene differences and offer a statistical test of whether the difference in variant counts is systematic, in addition to the single central-tendency value.
-
No correction for multiple comparisons is discussed, consistent with the paper not reporting formal p-values across its many dataset/metric/assembler comparisons.↳ Could also: A false discovery rate procedure such as Benjamini-Hochberg could also be applied if p-values across comparisons were computed — This would control for the increased chance of spurious findings when many comparisons (across datasets, thresholds, and assemblers) are considered jointly.
-
Performance was reported as curves (e.g., sensitivity vs. number of predicted transcripts) rather than as point estimates with dispersion.↳ Could also: Confidence bands around each curve, generated via bootstrap resampling, could also be reported — This would convey the uncertainty in sensitivity/precision at each threshold, complementing the point estimates already shown.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
Data side is clean, code side is broken. All five real-data accessions from the paper's Table 1 are public and complete (H1 rep1 41,419,823 read pairs vs the reported 41M), and the upstream TopHat2 2.1.1/Bowtie2 2.2.9 alignment ran cleanly (88.5% overall mapping, 75.3% concordant pairs, exit 0). The failure is in the paper's own subject software: Bayesembler aborts with Assertion *deletions_it == qualities.size() + 1 failed (assembler.cpp:1410, SIGABRT/exit 134) in both the stated publication release v1.1.1 and v1.2.0, under default and --no-pre-mRNA settings — corroborated by unanswered upstream issues #11–#15 reporting the identical assertion on independent datasets since 2015. The reproduction was careful rather than sloppy: the leading/trailing-D CIGAR root-cause hypothesis was explicitly tested and refuted by scanning all 98,435,535 reads, so the fault sits on the authors'/artifact side. Consequently no paper-stated value was comparable (q2 red), the central accuracy claim is unconfirmed (q7 red), and the Cufflinks/gffcompare run (Base Sn/Pr 31.3/69.9) is a non-equivalent proxy that speaks only to the input data's plausibility; q5 is yellow, not red, because nothing suggests the original figures were fabricated — only that the released code can no longer produce them.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.