WASP: a versatile, web-accessible single cell RNA-Seq processing platform.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Any deviation was negligible
- 🔴Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
WASP (Hoek et al. 2021) is a TOOL paper; our RU dataset SRP149565 = 3 mouse ddSEQ runs (ddSeq5/6/7). Compute RAN on «our HPC» (SLURM «job», node n127, 64 cores): all 3 public FASTQs (22.6 GB) downloaded to «infra» and processed. (1) C2 REPRODUCED within-tol: exact FASTQ read counts (345.0M/121.2M/99.7M = 565.9M barcode reads = 282.9M pairs) match ENA exactly and reconcile with the paper's 'roughly 283 million' as read PAIRS/spots. (2) C1 reproduced at the BARCODE-DEMULTIPLEXING stage only: WASP's demulti_umi.checkddSeq was re-run as a port VERIFIED byte-identical to the authors' source (commit cc3247c: linker1/linker2 + 96-barcode whitelist all match), yielding 349,773/247,218/232,937 distinct valid composite barcodes (36.9/38.4/36.9% of reads valid). The reported per-sample CELL counts (346/399/176) are NOT reproducible from public data because they depend on cDNA STAR-alignment + featureCounts gene-assignment + knee, and the cDNA mate is not in the public deposit (no knee on barcode-demux reaches those low counts; floor ~4000-6000 even at >=1000 reads/barcode). DID NOT attempt: STAR/featureCounts/UMI-tools cell counts (no cDNA), runtime/RAM (C3) and post-processing time (C4) (hardware-dependent). Honest partial reproduction: described well enough; data deposit materially incomplete for the headline numbers. All grades PROVISIONAL pending human review.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-18 ⛓ d09dfe6243d8
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-24
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper addresses the need for a single, freely available, user-friendly software platform that combines both pre-processing and post-processing of Drop-Seq-based (ddSEQ and 10x) single-cell RNA-seq data without requiring advanced bioinformatics or programming expertise.
- ★ WASP is a software platform for processing Drop-Seq-based scRNA-seq data generated with ddSEQ or 10x protocols, combining a Snakemake pre-processing pipeline with an R Shiny post-processing application. resource
- ★ WASP is, to the authors' knowledge, the only freely available software package that combines pre- and post-processing of ddSEQ- and 10x-based scRNA-seq data. finding
- ★ Due to its modular design, WASP's post-processing R Shiny application can be used with any similarly formatted gene expression matrix, including those generated by other pre-processing pipelines. finding
- ★ WASP calculates an automated barcode cutoff ('knee plot' method) to distinguish real cells from free-RNA-derived noise barcodes. method
- ★ WASP's post-processing supports normalization, clustering, differential expression analysis, and dimensionality reduction (PCA, t-SNE, UMAP, MDS) based on the Seurat R package. method
- WASP is distributed as a Docker container, via Conda for pre-processing, and as a standalone Windows version for post-processing requiring only a web browser. resource
- Separating pre-processing (Linux/high-RAM/compute cluster) and post-processing (cross-platform, low resource) modules provides flexibility to adapt to differing hardware/software requirements. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| scRNA-seq raw read pre-processing (demultiplexing, mapping, quantification) | not specified (software pipeline, generic ddSEQ-generated data) | none | gene expression matrix, quality metrics (JSON) | BioRad ddSEQ |
| scRNA-seq raw read pre-processing (demultiplexing, mapping, quantification) | not specified (software pipeline, generic 10x-generated data) | none | gene expression matrix, quality metrics (JSON) | 10x Genomics |
| Read quality control | not specified | none | sequence quality report | FastQC |
| Reference genome mapping | not specified (eukaryotic, e.g. human or mouse cells) | none | uniquely mapped, unmapped, multi-mapped reads | STAR |
| Feature extraction/quantification | not specified | none | reads assigned to single/multiple exonic features | featureCounts |
| UMI deduplication | not specified | none | deduplicated UMI counts per gene per barcode | UMI-tools |
| Post-processing analysis of gene expression matrix (clustering, differential expression, visualization) | generic gene expression matrix input (WASP-generated or external) | none | clusters, differentially expressed genes, t-SNE/UMAP/MDS/PCA plots | R Shiny / Seurat |
- – WASP provides a four-step Snakemake pre-processing workflow (FastQC, STAR, featureCounts, UMI-tools) generating demultiplexed gene expression matrices with quality metrics.
- – An automated 'knee plot' algorithm selects the barcode cutoff with the largest difference in a rank-normalized log-UMI score to separate real cells from free-RNA noise.
- – The R Shiny sample summary page reports read/barcode, STAR mapping, and featureCounts quality metrics in tables and stacked barplots for each processed sample.
- – The modular post-processing application accepts externally generated gene expression matrices, not only WASP pipeline output.
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a software methods paper describing WASP, a web-accessible scRNA-seq processing platform; it does not report a primary experimental study with between-group statistical comparisons. The paper describes algorithmic and computational approaches embedded in the tool—including a UMI-rank-based knee-point algorithm for barcode filtering, PCA-based dimensionality reduction, graph-based clustering, and Seurat-based differential gene expression—rather than applying these methods to a novel dataset and reporting inferential statistics. Results are presented as qualitative demonstrations of the pipeline's outputs (QC plots, UMAP/t-SNE embeddings, expression matrices).
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Seurat-based differential expression (specific test unspecified in the excerpted text) | Identification of differentially expressed genes across cell clusters in post-processing module | — | not stated |
| Rank-difference scoring algorithm (knee-point detection for barcode filtering) | Estimation of valid cell barcodes from UMI-count-ordered knee plot | — | not stated |
| Principal component analysis (PCA) | Dimensionality reduction prior to clustering and visualization | — | not stated |
-
Differential expression is delegated to Seurat without specifying which underlying test is used (Seurat supports Wilcoxon, likelihood-ratio, DESeq2 pseudo-bulk, and others)↳ Could also: Explicitly specify and expose the choice of DE test to the user (e.g., DESeq2 Wald test on pseudo-bulk aggregates, or MAST hurdle model for single-cell data) — Different DE tests carry different assumptions about count distributions and cell-level vs. sample-level variation; making the test explicit allows users to match the method to their experimental structure and aids reproducibility reporting
-
Barcode filtering uses a custom rank-difference knee-point scoring algorithm↳ Could also: Use established knee-point methods such as the inflection-point or total-UMI-fraction approaches implemented in packages like DropletUtils (emptyDrops or barcodeRanks) — DropletUtils emptyDrops uses a statistical test (Dirichlet-multinomial) to distinguish ambient RNA from true cells, which could complement or benchmark the heuristic rank-difference approach and is widely adopted in the field
-
Normalization of UMI counts is offered as an option but the specific method is not named in the excerpted text↳ Could also: Offer or document scran pooling-based normalization alongside the more common library-size (CPM/TPM-style) normalization — scran normalization accounts for the zero-inflation and size-factor heterogeneity common in single-cell data, and comparing normalization strategies can be informative when cell populations differ greatly in RNA content
-
Dimensionality reduction for visualization uses t-SNE, UMAP, and MDS↳ Could also: Also offer diffusion maps or force-directed graph embeddings (e.g., ForceAtlas2 as implemented in scanpy) — Diffusion maps are particularly informative for trajectory/pseudotime analyses, and some workflows use them as a complement to UMAP for continuous developmental processes
-
Highly variable gene selection is used to reduce noise before PCA, but the selection method is not specified↳ Could also: Apply and expose variance-stabilizing transformation (VST) or mean-variance trend modeling (as in scran or Seurat v3+) for HVG selection — In datasets with large dynamic range, VST-based HVG selection is less biased toward highly expressed genes than simple dispersion ranking, which can affect downstream clustering stability
-
The paper does not report a benchmarking comparison of WASP outputs against ground-truth or against other tools using quantitative metrics↳ Could also: Include a quantitative benchmark on a dataset with known cell-type composition (e.g., PBMC reference, simulated mixtures) reporting clustering accuracy (ARI, NMI) or DEG recovery — Quantitative benchmarking with reported metrics would allow readers to assess tool performance relative to alternatives such as Cell Ranger + Seurat or STARsolo and is common practice in scRNA-seq methods papers
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
scope.md — pmid-33736596 (WASP: web-accessible scRNA-seq processing platform)
Hoek et al. 2021, BMC Genomics. DOI 10.1186/s12864-021-07469-6. Code: github.com/andreashoek/wasp @ cc3247c (own code, authors' Snakemake + R/Shiny). Data: SRA SRP149565 / BioProject PRJNA473688 (ddSEQ, Mus musculus, mm10).
What WASP is
A TOOL paper. Two-part pipeline:
- Pre-Processing (Snakemake): FastQC → STAR (genome map, single-end on the
cDNA read) → featureCounts (subread, exon assignment) → custom ddSeq/10x
barcode demultiplexing (
demulti_umi.py) → UMI-toolscount --per-cell --per-gene→splitBarcodes.py(per-cell-barcode gene/UMI tables, cell count). - Post-Processing (R/Shiny): Seurat 3.1 + SingleCellExperiment + scater (normalise, cluster, t-SNE/UMAP, DE). Interactive, no fixed reported numbers.
Reported, pipeline-derived results for OUR dataset (SRP149565, ddSEQ, mm10)
(from Results / "Application" section of the paper)
| id | reported value | paper location | pipeline that produces it |
|---|---|---|---|
| C1 | ddSEQ5/6/7 valid barcodes = 346 / 399 / 176 (≈935/921 cells total) | Results, ddSEQ application | full pre-processing: STAR→featureCounts→demulti_umi(ddSeq)→umi_tools→splitBarcodes |
| C2 | "roughly 283 million reads" (ddSEQ dataset) | Results | raw read/pair count of SRP149565 |
| C3 | pre-processing ≈ 3.5 h, peak 28 GB RAM | Results | runtime/resource of the Snakemake run (environment-dependent) |
| C4 | post-processing of 935 cells < 3 min on a laptop | Results | Seurat post-processing (interactive) |
10x/Ciona (SRX4938158) and PBMC-2700 are SEPARATE demonstration datasets, NOT in this RU (our accession is SRP149565 only) → out of scope here.
IN SCOPE (attempted)
- C2 — read/pair counts of SRP149565: directly checkable from SRA/ENA + the downloaded FASTQs.
- C1 (barcode-demultiplexing half) — WASP's exact ddSeq barcode algorithm
(
demulti_umi.checkddSeq: linker1/linker2 Hamming search, 96-barcode whitelist with edit-distance ≤1, ACG/GAC anchor check, 18-bp composite barcode + 8-bp UMI) re-run faithfully on the REAL public barcode reads → number of reads with a valid ddSeq barcode + number of distinct valid cell barcodes + read/UMI support per barcode → barcode-level cell estimate, compared qualitatively to 346/399/176.
OUT OF SCOPE / BLOCKED (honest)
- C1 exact (346/399/176) — NOT reproducible from the public deposit. The
public SRA/ENA archive for all three runs loads only the technical
cell-barcode read (one 68–75 bp read per spot;
fastq-dump --split-spotyields a single barcode read;spots_with_mates=0). The cDNA (mRNA) mate needed for STAR alignment + featureCounts gene assignment is NOT publicly retrievable: it exists only as "Original" submitted files onsra-pub-src-4behind requester-pays "Use Cloud Data Delivery" (anonymous S3 = HTTP 403;prefetch --type allreturns only the normalized barcode run). No GEO series / processed count matrix exists (esearch gds → 0 hits). Because the reported cell count is the number of distinct valid barcodes among gene-assigned, aligned reads, it cannot be exactly reproduced without the cDNA → recorded as a data-access blocker (data_restrictedfor the cDNA component). - C3 / C4 — resource/runtime metrics (28 GB, 3.5 h, <3 min): environment- and hardware-dependent, not a deterministic pipeline output → not attempted.
- Post-processing Seurat clustering/DE: interactive, no fixed reported numbers.
Pipeline named per result
C1 → WASP Snakemake (STAR + subread/featureCounts + demulti_umi.py + UMI-tools + splitBarcodes.py). C2 → raw FASTQ read count. Reproduction of C1 is limited to the barcode-demultiplexing stage that the public data supports.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
This is a tool paper whose verifiable read-count claim reproduces cleanly (565.9M reads = 282.9M pairs ≈ the reported ~283M), and the exact ddSeq barcode-demultiplexing step re-runs faithfully on the real public reads. The core unreproduced numbers — per-sample cell counts 346/399/176 — are uncheckable because the cDNA mate is not anonymously retrievable (only the barcode read is archived; cDNA is behind requester-pays), so the failure sits on the data-availability side, not the authors'. No deviation in magnitude or direction was observed where comparison was possible, and there is no fabrication concern. Overall a defensible partial reproduction constrained by public-archive limits.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.