SPEAQeasy: a scalable pipeline for expression analysis and quantification for R/bioconductor-powered RNA-seq analyses.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Reported values were directly comparable
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- 🟡Could not use the authors’ exact input data
- 🟡A deviation arose in the data or preprocessing
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
SOFTWARE/METHODS paper (SPEAQeasy Nextflow RNA-seq pipeline). Described well enough to reproduce. CORE RESULT REPRODUCED 1:1: the pipeline runs end-to-end on its bundled hg38 test data (--small_test --reference hg38 --strand reverse --sample single, -profile local, conda toolchain) -- all 24 Nextflow processes COMPLETED exit 0 and produced the full documented output set (rse_gene/exon/jx/tx RangedSummarizedExperiment objects, raw/rpkm counts, featureCounts, kallisto tx, per-sample+merged genotype VCFs, FastQC, regtools junctions). Static artifacts: 740-SNV genotyping panel EXACT (C1). Honest discrepancies, neither a pipeline failure: shipped test FASTQs hold 250k reads vs the paper's stated 100k (C2, post-pub repo update); HISAT2/regtools versions drifted and the paper under-specifies regtools -- the pipeline actually requires the gpertea fork 0.5.33g for the -c junctions-count flag, not the stated 0.5.1 (C4). Three real blockers diagnosed+fixed to run on a Docker/Singularity-less HPC: FastQC font crash (font-capable openjdk=11 JRE + wrapper), git missing on PATH (main.nf:383), and the regtools 0.5.33g version requirement (built from source). NOT attempted: the SPEAQeasy-example companion vignette (separate repo, controlled-access dbGaP brain data, out of this RU's accession).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 53assessed: 2026-06-18 ⛓ 7d4b717ee0fb
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-25
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetExisting RNA-seq processing pipelines are difficult to install, share, or maintain and lack features important for bridging raw sequencing data to R/Bioconductor downstream analyses, motivating the development of a more portable, feature-complete, and reproducible pipeline.
- ★ SPEAQeasy is a portable, easy-to-install, Nextflow-powered RNA-seq processing pipeline that lowers the computational entry barrier for biologists/clinicians resource
- ★ SPEAQeasy produces RangedSummarizedExperiment R objects that integrate directly with Bioconductor downstream analysis packages resource
- ★ Reads are aligned with HISAT2 to a reference genome and pseudo-aligned with kallisto or Salmon to a reference transcriptome method
- Genes, exons, and exon-exon junctions are quantified using featureCounts and regtools method
- ★ SPEAQeasy performs RNA-based genotype calling with BCFtools that can be combined with DNA genotype data to detect and resolve sample swaps method
- SPEAQeasy supports human (hg19/hg38), mouse (mm10), and rat (rn6) reference genomes resource
- ★ SPEAQeasy is portable across computing environments (SGE, SLURM, local, Docker, AWS) via Nextflow configuration files finding
- ★ A vignette using BipSeq PsychENCODE bulk RNA-seq data demonstrates resolving sample identity issues and performing differential expression and gene ontology analysis with SPEAQeasy outputs finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| pipeline test run (alignment, quantification, QC) | provided test FASTQ samples for each reference organism/strandness combination | none | successful pipeline completion and run time | Nextflow |
| bulk RNA-seq differential expression analysis | human amygdala and subgenual anterior cingulate cortex (sACC), BipSeq PsychENCODE project | bipolar disorder vs neurotypical control diagnosis | differentially expressed genes, expression heatmap | edgeR/voom/limma |
| RNA-based genotype/variant calling | human RNA-seq samples | none | genotype calls at SNV sites used to detect sample swaps via correlation with DNA-based genotypes | BCFtools |
| ERCC spike-in quantification | RNA-seq samples with ERCC spike-ins | none | expected vs measured concentration for each ERCC transcript | — |
| gene ontology enrichment analysis | differentially expressed genes from BipSeq dataset | none | enriched biological processes/functional terms | clusterProfiler |
- – A typical SPEAQeasy test run completes in about 15 minutes (excluding first-run reference/annotation download and build time) ~15 min
- – SPEAQeasy produces a VCF file with genotype information at missense coding SNV sites with MAF > 30%, used for sample-swap detection 740 SNVs
- – Coverage bigWig files are normalized to a fixed read depth/length to allow cross-sample comparison 40 million mapped reads of 100bp
- – Differential expression analysis of BipSeq data identified genes above a significance threshold, visualized via heatmap clustered by sex, brain region, and diagnosis p < 0.2
- – SPEAQeasy generates per-sample plots comparing expected versus measured concentration for ERCC spike-in transcripts 92 ERCC transcripts
- count 740 SNVs (missense coding sites with MAF > 30% used in VCF genotype calling for sample-swap detection)
- count 92 ERCC transcripts (ERCC spike-in transcripts plotted for expected vs measured concentration)
- pvalue p < 0.2 (significance threshold used to select differentially expressed genes in example vignette)
- other 40 million mapped reads of 100bp (normalization standard for bigWig coverage files)
- other ~15 min (typical duration of a SPEAQeasy test run (post initial setup))
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a software pipeline paper describing SPEAQeasy, an RNA-seq processing workflow; formal statistical inference is confined to a demonstration vignette using the BipSeq PsychENCODE dataset comparing bipolar disorder cases to neurotypical controls across two brain regions. The demonstration pipeline normalizes counts with edgeR, corrects for heteroscedasticity with voom, fits linear models with limma, and applies an empirical Bayes shrinkage step to identify differentially expressed genes. Results are summarized visually (PCA plots, expression heatmaps) and through gene ontology enrichment, with genes selected at a nominal p < 0.2 threshold for illustration.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| limma empirical Bayes linear model (voom-transformed counts) | Differential expression analysis between bipolar disorder and neurotypical controls across amygdala and sACC (demonstration vignette) | — | not stated |
| Principal component analysis (PCA) | Exploratory assessment of variance attributable to total mapped reads, concordant map rate, sex, and brain region | — | na |
| Pairwise Pearson correlation (self-correlation matrix) | Sample identity / swap detection using SNV genotype calls from RNA-seq vs. pre-sequencing DNA genotypes | — | not stated |
| compareCluster (clusterProfiler gene ontology over-representation / enrichment) | Functional annotation of significantly differentially expressed genes (p < 0.2) | — | not stated |
-
Differential expression was assessed using edgeR normalization followed by voom transformation and limma linear modeling with empirical Bayes shrinkage↳ Could also: DESeq2 (negative binomial GLM with Wald or likelihood-ratio test) could also be applied to the same count matrix — DESeq2 and edgeR/voom/limma are both well-validated for RNA-seq; DESeq2 uses a parametric shrinkage estimator for dispersion and log2 fold-change, while voom models the mean-variance trend; comparing both can serve as a sensitivity check and is common in high-profile studies
-
Genes were selected for downstream visualization using a nominal threshold of p < 0.2 (no multiple-testing correction named)↳ Could also: Applying a Benjamini-Hochberg FDR threshold (e.g., q < 0.05 or q < 0.1) to the full set of tested genes would also control the expected proportion of false discoveries — With tens of thousands of genes tested simultaneously, FDR control is a standard practice; a nominal p-threshold without explicit correction may retain a larger number of false positives, which is acceptable for illustration but worth noting when interpreting biological conclusions
-
Exploratory structure in gene expression was visualized using PCA on the first ten principal components↳ Could also: UMAP or t-SNE could also project the high-dimensional expression space for visualization, particularly to reveal nonlinear sample groupings — PCA is linear and interpretable; UMAP/t-SNE can reveal cluster structure that PCA may compress into higher components, and are increasingly common in bulk RNA-seq QC alongside PCA
-
Sample identity verification used pairwise Pearson correlation between RNA-seq-derived and pre-sequencing SNV genotype profiles↳ Could also: Concordance-based metrics (e.g., identity-by-descent estimated with KING or a chi-squared test of genotype concordance per SNV) could also be used to flag swaps — Correlation is intuitive and widely used for this purpose; concordance or IBS-based methods can provide formal statistical tests and are robust to missing genotype calls at individual SNV positions
-
Functional enrichment of differentially expressed genes was assessed with clusterProfiler's compareCluster using gene ontology terms↳ Could also: Gene set enrichment analysis (GSEA) on the full ranked gene list, or fgsea, could also be applied without requiring a hard significance cutoff to define the gene set — Over-representation analysis (ORA, as used here) depends on the chosen significance threshold to define the input list; GSEA uses the full ranked list and can detect enrichment even when individual genes do not pass a cutoff, making it less sensitive to the choice of p-threshold
-
Dispersion of gene expression across samples is not summarized with any reported statistic (no SD, SEM, or CI in the provided results)↳ Could also: Reporting log2 fold-changes with 95% confidence intervals (available from limma's topTable) alongside p-values would also characterize effect magnitude and its uncertainty — Effect sizes with confidence intervals convey both the direction and precision of expression differences, complementing p-values especially when sample sizes differ across groups or when comparing results across studies
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — PMID 33932985 (SPEAQeasy)
Paper: SPEAQeasy: a scalable pipeline for expression analysis and quantification
for R/bioconductor-powered RNA-seq analyses. Eagles et al., BMC Bioinformatics 2021.
PMCID PMC8088074 · DOI 10.1186/s12859-021-04142-3.
Repo (authors' own, P16 N/A — it IS the tool): https://github.com/LieberInstitute/SPEAQeasy
(cloned commit 4019825 on «infra»).
What kind of paper
A software/methods paper describing a Nextflow RNA-seq pipeline (FASTQ → quality
control → HISAT2 alignment → featureCounts/regtools/salmon/kallisto quantification →
bcftools genotyping → R/Bioconductor RangedSummarizedExperiment objects). It ships
its own bundled test data (human/mouse/rat, single/paired, fwd/rev/unstranded).
There are very FEW hard numeric claims (it is not a data-analysis paper). The reproducible computational anchors are:
In scope (pipeline-derived / shipped-artifact, attempted)
| id | result | reported | how reproduced | pipeline |
|---|---|---|---|---|
| C1 | genotyping SNV panel = 740 missense SNV sites, MAF>30% | "a list of 740 single nucleotide variant (SNV) missense coding sites with MAF > 30%" (Results: SPEAQeasy output files) | count rows of shipped Annotation/Genotyping/common_missense_SNVs_{hg38,hg19}.bed |
repo static artifact (input to VariantCalls/bcftools) |
| C2 | test samples subsetted to 100,000 reads | "Each was then subsetted to 100,000 reads." (Implementation: Test samples) | count reads in shipped test/human/single/reverse/*.fastq.gz |
repo static artifact |
| C3 | pipeline runs on its own test data & produces the documented outputs (delivers-promised) | qualitative; "a typical test run may complete in about 15 min" (Results: Configuring SPEAQeasy) | run SPEAQeasy --small_test --reference hg38 end-to-end on «our HPC» (-profile local, conda toolchain), inspect outputs (BAMs, gene counts, junctions, salmon tx, genotype VCF, RSE objects) |
full SPEAQeasy Nextflow pipeline |
| C4 | tool versions reported in Methods | HISAT2 2.1.0, salmon 1.2.1, kallisto 0.46.1, Subread/featureCounts 2.0.0, bcftools 1.10.2, FastQC 0.11.8, Trimmomatic 0.39, regtools 0.5.1 | compare to repo conf/docker.config + install_software.sh pins |
repo metadata |
Out of scope (not attempted)
- The runtime "~15 min" claim is hardware/cache-dependent (first run downloads & builds the full hg38 reference) — not a 1:1 numeric target; reported only as context.
- The downstream SPEAQeasy-example vignette (sample-swap detection, bipolar BipSeq PsychENCODE case study at research.libd.org/SPEAQeasy-example) is a separate companion repo with its own (controlled-access dbGaP-class) brain data; the paper's own deposited reproducible unit is the bundled test data + the pipeline. The BipSeq individual-level data is not part of this RU's accession (sra:SRS7176970).
Notes
--small_testonly swaps the input FASTQs to the bundled test files; it still pulls & builds the full hg38 GENCODE reference + HISAT2 index (main.nf L274–516).- No docker/singularity on «our HPC» → run via
-profile localwith a conda toolchain, command paths repointed fromconf/command_paths_long.config.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
This is a methods/software paper (SPEAQeasy Nextflow RNA-seq pipeline); its one hard numeric artifact — the 740-site missense-SNV genotyping panel (MAF>30%) — reproduces exactly in both shipped hg38/hg19 BEDs, and 6/8 tool versions match the paper. The only deviations are on the authors' maintained-repo side but benign: test FASTQs now carry 250k reads vs the paper's stated 100k, and HISAT2/regtools versions drifted upward — both consistent post-publication updates, with no fabrication signal. Severity is moderate only because the read-count claim factually fails 1:1, and the core-claim grade is limited because the full end-to-end executability run (C3) is still pending on «our HPC» rather than failed.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.