pyrpipe: a Python package for RNA-Seq workflows.
The main results reproduced: recomputed values matched the published ones within tolerance.
- Nothing in this column.
- 🟡Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
REPRODUCED (1:1). pyrpipe is a tool paper; the reproducible content is Case Study 2 (COVID-19 monocyte RNA-Seq, SRP287810). Ran the repo's exact Snakemake recipe on «our HPC» compute nodes: salmon 1.4.0 decoy-aware index built verbatim from prepare_data.sh (GENCODE v36 transcripts + GRCh38 no-alt genome + SARS-CoV-2 RefSeq decoys; 196 decoys, default k=31), then salmon quant -l A on all 29 RNA-Seq runs (mapping rate 48-80%), then merged to transcript/gene TPM matrices. Results match the authors 1:1: R1 29 samples = exact; R2 60240 genes x 29 (the shipped MOG matrix itself has 60240 rows; the '60241' filename label is +1); R3 232117 GENCODE v36 tx -> 231288 after salmon dedup (829 duplicate_clusters); R4 method exact; R5 (strongest 1:1) reproduced gene TPM matrix vs shipped MOG matrix: mean Pearson 0.99998, mean Spearman 0.99982, 99.48% of ~1.75M values within 1%, median |diff| 0. NOT attempted: downstream GO-enrichment (R6/R7) - interactive MetaOmGraph gene-ranking + external GO tool, not specified in repo, out of scope. Prior requeue blocker (exhausted «infra» quota) is resolved; all compute ran inside SLURM jobs on compute nodes. Full TPM matrices on «infra» (SHA256 in reproduction/outputs/MANIFEST_beegfs.txt); small derived results pulled to «host».
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-18 ⛓ b7609f5f88fa
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-24
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThere is a need for a framework that enables fast, efficient, manageable, and reproducible RNA-Seq analysis pipelines, addressing the limitations of plain Bash/Perl/Python scripting for bioinformatics workflows.
- ★ pyrpipe enables development of flexible, reproducible, and easy-to-debug RNA-Seq computational pipelines purely in Python, in an object-oriented manner resource
- ★ pyrpipe provides high-level APIs to 11 popular RNA-Seq analysis tools grouped into cohesive modules (sra, mapping, alignment, quant, qc, tools) resource
- ★ pyrpipe supports checkpoints, session saving, dry-run mode, and skipping of already-completed steps to enable flexible execution across computing environments method
- ★ pyrpipe produces detailed logging (JSON format), diagnostic reports, benchmark comparisons, and MultiQC reports via the pyrpipe_diagnostic tool method
- ★ pyrpipe enhances reproducibility by logging MD5 checksums of pipeline scripts and input files, tool versions, and supporting YAML-based parameter management method
- pyrpipe can be integrated into workflow management systems such as Snakemake, NextFlow, or Toil for scaling pipelines across clusters method
- ★ pyrpipe scales to processing very large RNA-Seq datasets, demonstrated on 17,328 GTEx samples, with results concordant with the official GTEx pipeline finding
- pyrpipe can be used for de novo transcriptome assembly and identification of putative unannotated (orphan) genes finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| bulk RNA-seq processing (alignment, assembly, quantification) | human, 17,328 GTEx samples across non-diseased tissues | none | transcript/gene-level TPM expression | STAR, Stringtie, orfipy, Mikado, Taco, Salmon; run on PSC Bridges HPC |
| bulk RNA-seq quantification within workflow managers (Snakemake, NextFlow) | human circulating monocytes, COVID-19 patients (n=20) vs healthy individuals (n=9), 29 samples (SRP287810) | COVID-19 disease status; hydroxychloroquine treated vs untreated | transcript and gene-level TPM, differential expression via Mann-Whitney test | Salmon selective alignment; analyzed via MetaOmGraph |
| de novo transcriptome assembly | Zea mays B73 cultivar, 10 diverse samples (different tissues/development stages) | none | number and length of assembled transcripts, homology to UniProt proteins, GC content | reference-guided assembly using new B73 genome; UniProt homology search |
- – Median TPMs of annotated genes computed by pyrpipe pipeline closely match GTEx portal values for visceral and subcutaneous adipose tissue Pearson correlation 0.97
- ▲ Nine of fourteen genes most overexpressed in healthy monocytes are involved in neutrophil chemotaxis, including six CCL/CXL-type chemokines GO:0030593, Bonferroni-corrected P = 5.229E-10
- ▲ Twelve of fourteen genes highly expressed in COVID-19 monocytes but not healthy individuals participate in immune effector processes, including nine defensins/immunoglobulins GO:0002252, Bonferroni-corrected P = 3.161E-12
- – De novo assembly of Zea mays RNA-Seq identified a total of 57,916 distinct transcripts 57,916 transcripts
- – 38,881 assembled maize transcripts were homologous to UniProt proteins; 6,306 showed no similarity to any protein in the database (potential ncRNAs, noise, or orphan genes) 38,881 vs 6,306 transcripts
- ▼ Non-homologous maize transcripts are shorter on average than conserved transcripts mean length 1290 nt vs 1981 nt
- correlation Pearson correlation 0.97 (comparison of median TPMs between pyrpipe pipeline and GTEx portal pipeline for adipose tissue)
- count 17,328 RNA-Seq samples processed (GTEx case study scaling demonstration)
- count 29 RNA-Seq samples (20 COVID-19, 9 healthy) (COVID-19 monocyte case study sample size)
- pvalue Bonferroni-corrected P = 5.229E-10 (neutrophil chemotaxis GO enrichment in healthy-overexpressed genes)
- pvalue Bonferroni-corrected P = 3.161E-12 (immune effector process GO enrichment in COVID-19-overexpressed genes)
- count 57,916 distinct transcripts assembled (Zea mays de novo transcriptome assembly total output)
- count 38,881 homologous / 6,306 non-homologous transcripts (UniProt homology classification of assembled maize transcripts)
- mean 1290 nt vs 1981 nt mean transcript length (non-homologous vs conserved transcript length comparison in maize)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a software methods paper introducing pyrpipe, a Python package for RNA-Seq pipelines; statistical analyses appear only within three illustrative case studies. Case Study 1 uses Pearson correlation to validate agreement between the pyrpipe pipeline and the GTEx reference pipeline on median log-TPM values. Case Study 2 applies Mann–Whitney U tests with Benjamini–Hochberg correction to identify differentially expressed genes in COVID-19 versus healthy monocytes, followed by Bonferroni-corrected GO enrichment via ToppGene. Case Study 3 reports descriptive statistics (mean transcript length, GC content) for a de novo Zea mays transcriptome assembly.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Pearson correlation | Comparison of median log-TPMs between pyrpipe pipeline and GTEx portal pipeline for two adipose tissue types (Case Study 1, Figure 2) | Not stated (number of annotated genes compared; r reported as 0.97, written as '0.97%' in text) | not stated |
| Mann–Whitney U (non-parametric) | Differential expression between COVID-19 (n=20) and healthy (n=9) monocyte samples; filter: >2-fold TPM change and BH-adjusted p < 0.002 (Case Study 2, Figure 3) | 29 samples total (20 COVID-19, 9 healthy); test applied across >60,000 genes | not stated |
| GO over-representation analysis with Bonferroni correction (via ToppGene) | Functional annotation of 14 genes most highly overexpressed in healthy vs COVID-19 monocytes and 14 genes most highly overexpressed in COVID-19 vs healthy monocytes (Case Study 2) | 14 genes per direction | not stated |
| Mutual Information | Estimation of non-linear association of other genes with CCL2 across COVID-19 and healthy samples (Case Study 2, Figure 3A) | 29 samples | not stated |
| Descriptive statistics (mean) | Transcript length and GC content summary for de novo Zea mays transcriptome assembly (Case Study 3) | 57,916 transcripts total | na |
-
Differential expression in Case Study 2 was identified by combining a fold-change threshold (>2-fold TPM) with Mann–Whitney U tests applied to TPM values↳ Could also: A count-level differential expression framework such as DESeq2 or edgeR could also have been applied directly to raw or estimated read counts — Count-based models explicitly account for overdispersion in RNA-Seq data and produce shrinkage-stabilised fold-change estimates; they are a standard approach when the primary goal is a statistically principled ranked gene list from sample-level data
-
Pipeline agreement in Case Study 1 was summarised with Pearson correlation of median log-TPM values across two tissue types↳ Could also: Spearman rank correlation or a Bland–Altman (limits-of-agreement) analysis could also have been used — Spearman correlation is robust to outliers and non-normality in log-TPM distributions; Bland–Altman analysis additionally characterises the magnitude and pattern of systematic bias across the range of expression values, which is informative for pipeline benchmarking
-
The COVID-19 vs healthy comparison involved markedly unequal group sizes (n=20 vs n=9)↳ Could also: Permutation-based differential expression testing or bootstrap resampling could also have been used — Permutation and bootstrap methods make no parametric distributional assumption and remain valid with small, unequal group sizes; they complement the Mann–Whitney test and can provide empirical p-values directly from the observed data structure
-
GO enrichment was assessed using Bonferroni correction on sets of 14 pre-selected genes per direction↳ Could also: Gene Set Enrichment Analysis (GSEA) on the full ranked gene list, or Fisher's exact test with FDR correction, could also have been applied — GSEA uses the complete ranked list rather than a fold-change threshold, reducing sensitivity to the chosen cutoff; Fisher's exact test with FDR is a widely used alternative to Bonferroni for over-representation analysis when gene lists are small and Bonferroni may be conservative
-
Transcript length and GC content in Case Study 3 were summarised by mean values alone↳ Could also: Median and interquartile range (IQR), or kernel density estimates, could also have been reported — Transcript length distributions are typically right-skewed; median and IQR are more robust summaries of central tendency and spread for skewed data, and density plots convey the full distributional shape rather than a single point estimate
-
Non-linear gene–gene association with CCL2 was estimated using Mutual Information↳ Could also: Spearman rank correlation or distance correlation could also have been used to capture non-linear dependencies — Spearman correlation is computationally simple and widely interpretable; distance correlation has a well-characterised null distribution facilitating formal inference; both are standard alternatives for non-linear association in small-sample transcriptomic exploratory analyses
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-34085037 (pyrpipe: a Python package for RNA-Seq workflows)
Singh U, Li J, Seetharam A, Wurtele ES. NAR Genom Bioinform 2021. PMC8168212. DOI 10.1093/nargab/lqab049. Repo: https://github.com/urmi-21/pyrpipe (HEAD commit f999f9d8689c43aabdd5b621511e4299cbba2e28).
What this paper is
A software/tool paper. pyrpipe is a Python package that wraps standard RNA-Seq command-line tools (SRA-Tools, Trim Galore/BBduk, STAR/HISAT2, Salmon/Kallisto, StringTie, samtools) into scriptable, reproducible workflows, and integrates with Snakemake/Nextflow. The paper's quantitative content is two demonstration case studies; the relevant data accession for this RU is SRP287810, used in Case Study 2 (COVID-19 monocyte RNA-Seq).
Per the brief (P16): applying the authors' shipped pipeline (or the underlying third-party tool, Salmon) to the paper's own data is a fully valid reproduction.
In scope (pipeline-derived, reproduced here)
The Case Study 2 pipeline is fully specified in the repo under
case_studies/Covid_RNA-Seq/ (Snakemake + Nextflow, identical logic):
- Download 29 RNA-Seq runs from SRA accession SRP287810
(
runids.txt: SRR12850397–SRR12850416 + SRR12850435–SRR12850443). - Reference build (
Snakemake/prepare_data.sh): GENCODE v36 human transcripts + GRCh38 genome (GCA_000001405.15_GRCh38_no_alt_analysis_set)- SARS-CoV-2 genome (taxon 2697049) → concatenated
human_tr_gen_decoy.fastawithdecoys.txt(genome + viral seq names = decoys).
- SARS-CoV-2 genome (taxon 2697049) → concatenated
- Salmon 1.4.0 decoy-aware index (selective alignment), default k=31.
- Quantify each run with pyrpipe
sra.SRA(id).quant(salmon)→ per-samplequant.sf(transcript-level TPM). - Merge (
merge.py/ Snakefilemergerule) → transcript-level (results_TPM_tx.tsv) and gene-level (results_TPM_gene.tsv) TPM matrices; gene-level = sum TPM over GeneID, version stripped, annotated viaEns_gene_metadata.txt.
Reproducible numeric targets:
- R1: 29 RNA-Seq samples quantified (SRP287810). [paper Results, Case Study 2]
- R2: gene-level TPM matrix dimensions = 60241 genes × 29 samples
[repo artifact
MOGproject-monocytes-60241genes-29samples-HCQtreat-2021-1-17.zip] - R3: transcript-level matrix row count = N transcripts in GENCODE v36 (verified at runtime, not asserted).
- R4: per-sample Salmon mapping rate (sanity / QC, no paper target).
- R5 (strongest 1:1, if MOG matrix extractable): per-gene/per-sample TPM values vs the shipped merged matrix inside the MOG project zip.
Out of scope (NOT attempted here, or partial)
- Downstream MetaOmGraph (MOG) analysis producing the GO-enrichment findings (neutrophil chemotaxis GO:0030593 Bonferroni P=5.229E-10, 9/14 genes; immune effector process GO:0002252 Bonferroni P=3.161E-12, 12/14 genes). These come from manual/interactive MOG steps (gene ranking + external GO enrichment tool); the exact DE/ranking criteria and GO tool are not fully specified in the repo → treated as PARTIAL / downstream, recorded as claims but not regenerated 1:1.
- Case Study 1 (Arabidopsis, different accession) — not this RU's data.
- Benchmark/runtime statements, software-engineering claims (test coverage, logging, API design) — narrative, not a numeric pipeline output.
- Wet-lab / clinical sample provenance (COVID vs healthy labelling) — external.
Pipeline named per result
All in-scope results (R1–R5) derive from the Salmon selective-alignment
quantification pipeline driven by pyrpipe, exactly as shipped in
case_studies/Covid_RNA-Seq/.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.