nf-rnaSeqCount: A Nextflow pipeline for obtaining raw read counts from RNA-seq data.
The main results reproduced: recomputed values matched the published ones within tolerance.
- ✓Same input data as the authors
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
REPRODUCED. Tool/methods paper: the nf-rnaSeqCount Nextflow pipeline. Reproduced its core promise by running the actual published tool (paper-era DSL1 commit 5a3276488f56) on its own test data (GSE111073/SRP133401, 3-sample subset) + Ensembl GRCh38 r110, end-to-end on the «our HPC»/«infra» SLURM HPC via its own Singularity containers (apptainer 1.4.5). Confirmed C1 (both featureCounts AND htseq-count emit a 62,754-gene x 3-sample integer count matrix; the two counters correlate at Pearson 0.995-0.997), C2 (valid genes x samples integer DE input), C3 (runs end-to-end on a 4th HPC). Found + fixed a genuine repo bug along the way: nextflow.config binds the genome/genes/data/out dirs into the container but NOT the alignment output dir, so featureCounts -- which is handed absolute BAM paths via sample_bams.txt rather than nextflow-staged BAMs like htseq -- aborts with 'ERROR: invalid parameter: ..._Aligned.out.bam'; fixed by adding '-B <workdir>' to singularity.runOptions (an env SINGULARITY_BIND does not work because nextflow's nxf_launch strips the env with env -). NOT attempted (out of scope, hardware-dependent): Fig-3 wall-clock runtimes and the $9.39 AWS cost. Verdict is provisional and human-auditable (claims.tsv + agreement.json + AUDIT.md).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-19 ⛓ 93dc01fc2333
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-25
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper aims to determine whether a Nextflow/Singularity-based pipeline (nf-rnaSeqCount) can produce a portable, reproducible, and scalable way to generate raw RNA-seq read counts, reducing analysis complexity and time compared to existing tools.
- ★ nf-rnaSeqCount is a portable, reproducible Nextflow pipeline that maps RNA-seq reads to a reference genome and quantifies gene abundance for differential expression analysis resource
- ★ Combining Nextflow (workflow management) with Singularity (containerisation) achieves pipeline portability and reproducibility across HPC and cloud platforms method
- ★ nf-rnaSeqCount can distribute tasks across multiple nodes (parallel execution) in addition to multi-threading, unlike Rsubread which is multi-threaded on a single node only finding
- ★ nf-rnaSeqCount (STAR) outperforms Rsubread (align) in read alignment speed finding
- ★ nf-rnaSeqCount uses more memory and CPU resources than Rsubread for indexing and alignment tasks finding
- The pipeline was successfully tested and validated on Wits cluster, UCT eResearch HPC, and AWS using real SSc patient RNA-seq data finding
- Running the full nf-rnaSeqCount analysis on AWS for the SSc dataset cost approximately $9.39 finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| RNA-seq pipeline validation (alignment + read counting) | Black South African SSc patients and healthy controls (human blood/tissue, Frost et al. 2019 study) | none (disease vs healthy comparison) | raw gene read count matrices, pipeline execution success | Wits Computing cluster (SLURM/PBS), UCT eResearch HPC (SLURM), AWS EC2 m4.10xlarge |
| Benchmarking: reference genome indexing | GEO GSE111073 breast cancer RNA-seq samples (21 samples: 10 walnut-consuming, 11 control) | none (tool comparison) | time, memory, CPU usage | Wits Computing cluster; STAR/Bowtie2 (nf-rnaSeqCount) vs Rsubread index |
| Benchmarking: read alignment | GEO GSE111073 breast cancer RNA-seq samples | none (tool comparison) | time, memory, CPU usage | Wits Computing cluster; STAR (nf-rnaSeqCount) vs Rsubread align |
| Benchmarking: read counting/quantification | GEO GSE111073 breast cancer RNA-seq samples | none (tool comparison) | time, memory, CPU usage | Wits Computing cluster; htseq-count/featureCounts (nf-rnaSeqCount) vs Rsubread |
| Quality control (FastQC, Trimmomatic) | RNA-seq FASTQ reads (paired- and single-ended) | none | read quality metrics, trimmed reads | FastQC, Trimmomatic |
- ▼ nf-rnaSeqCount (STAR) completed read alignment faster than Rsubread (align): 31 minutes vs 215 minutes ~6.9-fold faster
- – Genome indexing time was similar between tools: 66 minutes (STAR/Bowtie2) vs 71 minutes (Rsubread index) 66 vs 71 min
- ▲ nf-rnaSeqCount used more memory and CPU for indexing than Rsubread: 35GB/825% CPU vs 16GB/99% CPU 35GB vs 16GB; 825% vs 99% CPU
- – For alignment, nf-rnaSeqCount used 30GB/648% CPU while Rsubread used 18GB/1088% CPU 30GB/648% vs 18GB/1088%
- – Total estimated AWS cost for running nf-rnaSeqCount on the SSc dataset was $9.39 over ~4 hours $9.39
- – nf-rnaSeqCount pipeline was successfully executed on Wits cluster, UCT HPC, and AWS using SSc patient data
- fold_change 215 min / 31 min ≈ 6.9-fold (Read alignment speed: Rsubread vs nf-rnaSeqCount (STAR))
- other 66 minutes (STAR/Bowtie2) vs 71 minutes (Rsubread) (Genome indexing time comparison)
- other 35 GB memory, 825% CPU (nf-rnaSeqCount) vs 16 GB, 99% CPU (Rsubread) (Resource usage during indexing)
- other 30 GB memory, 648% CPU (nf-rnaSeqCount) vs 18 GB, 1088% CPU (Rsubread) (Resource usage during alignment)
- other $9.39 total estimated cost (AWS run cost for ~4-hour SSc dataset analysis (m4.10xlarge at $2.22/hr, SSD $0.11/GB-month, 1000GB))
- count 21 samples (10 walnut-consuming, 11 control) (Benchmarking dataset GSE111073 breast cancer RNA-seq samples)
- other 40 virtual CPUs, 160 GB memory, 1000GB EBS (AWS EC2 m4.10xlarge instance specifications used for testing)
- other 48 GB memory, 12 CPUs allocated per task (Wits Computing cluster resources for benchmarking)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a bioinformatics software/methods paper describing the nf-rnaSeqCount pipeline (built on Nextflow and Singularity) for RNA-seq alignment and quantification. The paper reports a single benchmarking comparison between nf-rnaSeqCount and the Rsubread package, presenting time, memory, and CPU usage for genome indexing, read alignment, and read counting on one dataset (21 samples), with results reported as point measurements (e.g., minutes, GB, % CPU) rather than through formal statistical hypothesis testing.
-
Pipeline benchmarking (time, memory, CPU) was performed as a single run per tool per task on one dataset, with results reported as point estimates.↳ Could also: Repeating each benchmarking task multiple times (technical replicates of the same run) and reporting mean ± SD or a range — This would convey how much run-to-run variability exists in resource usage on shared HPC/cloud systems, which can fluctuate due to contention with other jobs, and would let readers judge whether observed differences between tools exceed typical run-to-run noise.
-
The comparison between nf-rnaSeqCount and Rsubread performance metrics is presented descriptively (e.g., '31 vs 215 minutes') without a formal statistical test.↳ Could also: A paired comparison test (e.g., paired t-test or Wilcoxon signed-rank test) across multiple benchmarking replicates or datasets — This would allow a quantitative statement about whether the performance difference between tools is unlikely to be due to chance, complementing the raw descriptive numbers already provided.
-
Benchmarking was conducted using a single dataset (GSE111073, 21 samples) run once through each pipeline.↳ Could also: Benchmarking across multiple independent datasets of varying size — This would help characterize how the relative performance and resource usage of the two tools scales with dataset size and composition, supporting broader generalizability of the comparison.
-
Cost estimation for AWS execution was presented as a single calculated dollar figure based on one observed runtime (~4 hours).↳ Could also: Reporting a range or repeated-measurement estimate of runtime (and corresponding cost range) — Since cloud runtimes can vary between executions, a range would give readers a more robust expectation of real-world cost rather than a single deterministic value.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-35574063
Paper: Mpangase et al. (2021) nf-rnaSeqCount: A Nextflow pipeline for obtaining raw read counts from RNA-seq data. S Afr Comput J 33(2). DOI 10.18489/sacj.v33i2.830.
Nature of the paper: This is a software/methods (tool) paper. It presents a
Nextflow pipeline (nf-rnaSeqCount) whose stated purpose is to take raw RNA-seq
reads + a reference genome/annotation and produce raw gene read-count matrices
ready for downstream DE analysis (DESeq2/edgeR). Most of the paper's "numbers" are
benchmark runtimes measured on the authors' specific HPC — these are
hardware-dependent and not 1:1 reproducible on different hardware.
Code: https://github.com/phelelani/nf-rnaSeqCount (MIT, public, not archived).
- Current
masterHEAD =a4bb1f9e78d68babcede007e7bd5f24cdee6bd66(2023-08-02, "Switching to Nextflow DSL2") — a post-paper rewrite. - Paper-era version (2021): latest 2021 commit
5a3276488f56(2021-09-28 "Memory fix"). We pin to the 2021 version for fidelity to the paper, and note the DSL2 master as an alternative.
Data: GEO GSE111073 → SRA SRP133401. 21 paired-end human RNA-seq samples (Illumina HiSeq 1500), 10 walnut-consuming + 11 control breast-cancer patients. ~31 GB SRA. (This is the Hardman walnut/breast-cancer dataset; the paper re-uses it purely as a test input to exercise the pipeline.)
Pipeline steps (the tool)
prep.Containers— pull Singularity images (FastQC, STAR, featureCounts, HTSeq, Trimmomatic, MultiQC, Bowtie2)prep.Indexes— STAR genome index fromgenome.fa+genes.gtfrun.ReadQC(FastQC, optional)run.ReadTrimming(Trimmomatic, optional)run.ReadAlignment(STAR) → BAMsrun.ReadCounting(featureCounts and htseq-count) →gene_counts_final.txtrun.MultiQC(optional)
IN SCOPE (pipeline-derived, will attempt)
| id | result | pipeline | how to reproduce |
|---|---|---|---|
| C1 | The pipeline produces raw gene read-count matrices (gene_counts_final.txt) from RNA-seq input via STAR→featureCounts/htseq-count |
nf-rnaSeqCount | Run pipeline on GSE111073 (subset/full) + GRCh38; verify a genes×samples integer count matrix is emitted by both featureCounts and htseq-count |
| C2 | Output is a valid DE-analysis input (raw integer counts, genes as rows, samples as columns) | nf-rnaSeqCount | Inspect produced matrix: dimensions, integer counts, non-trivial signal, two counters agree in structure |
| C3 | Pipeline runs end-to-end on an HPC (SLURM) using its containers | nf-rnaSeqCount | Successful SLURM execution on «our HPC» (one of the "three HPC platforms" portability claim, partially) |
The core "delivers what it promises" test = C1/C2: does the tool actually turn the paper's own data into the count matrices it claims to produce?
OUT OF SCOPE (not attempted / not 1:1 reproducible — stated, not graded as mismatch)
- Figure 3 runtimes (Indexing 66 min / Alignment 31 min / Counting 235 min for nf-rnaSeqCount; Rsubread 71/215/6 min) — hardware-, CPU-count-, and IO-dependent on the authors' HPC. We may report our own wall-times as informational context only, never as a claim "match".
- AWS cost = $9.39 (Section 2.4) — arithmetic derived from runtime × instance price; not an independent computational result.
- SSc (Frost et al. 2019) application data — a different dataset, the downstream use-case, not the pipeline test data.
- Wet-lab / clinical aspects of GSE111073 (walnut study) — not this paper's work.
Reproduction plan
All heavy compute on «our HPC» (SLURM), all data on «infra». Quick-minimum = run the counting workflow on a manageable subset of GSE111073 samples + GRCh38 and confirm valid count matrices from both counters (C1/C2). Then scale toward more/all 21 samples as feasible.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.